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Executive Summary 


The Literacy Design Collaborative (LDC) was created in 2009 to support teachers in 
implementing Common Core State Standards (CCSS) and embedding literacy skill development 
throughout content area curriculum. Engaged in the evaluation of LDC tools since June 2011, 
UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is 
the independent evaluator for LDC’s current federally funded i3 validation grant. CRESST’s 
evaluation study is using multiple data sources and a quasi-experimental design (QED) to 
examine LDC implementation and impact in two cohorts of schools in two large, urban school 
districts. 


This report presents the results on implementation of LDC in the New York City 
Department of Education during the third year of the intervention, and the impact of the 
program across multiple years. The study schools serve largely Hispanic and Black populations, 
with a high proportion of students from economically disadvantaged backgrounds, and many 
English language learners (ELLs). As of 2018-2019, participating schools included 13 from 
Cohort 1, which began implementation during the 2016-2017 school year, and 23 from 
Cohort 2, which commenced at the beginning of the 2017-2018 school year. Our primary 
impact analyses, presented in this report for the first time, pool teachers from both cohorts to 
measure their impact after participating in LDC for 2 consecutive years (2017-2018 student 
outcomes for Cohort 1 and 2018-2019 student outcomes for Cohort 2). 


The CRESST evaluation addresses research questions in three major areas: 
e Program Characteristics and Implementation 

e Contextual Factors and Implementation 

e Program Impacts 


The findings draw on multiple data sources and methods across the 5 years of the study. 


These include surveys of teachers, teacher leaders, and administrators; analysis of the quality of 
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performance tasks called LDC modules, which are a central manifestation of LDC practice; 
participant interactions with LDC CoreTools, the electronic platform through which teachers 
access LDC professional development resources (online courses, existing LDC modules, module 
templates, and support for module development); data on teacher attendance, meeting 
lengths, and coach/teacher leader calls captured in professional learning communities (PLC) 
reflection forms; LDC administrative records capturing participant attendance at PLC sessions 
and professional development (PD) offerings for teacher leaders and administrators; and 
administrative data on students and teachers including class rosters, student demographics, 
and student performance on state standards-based assessments. We begin with the overall 
findings and then summarize participants’ perspectives on key LDC components, intermediate 
effects on teachers’ instructional strategies and practice, and effects on student outcomes. 
Detailed evidence with regard to key LDC activities, supports, and pedagogical impacts help to 
explain the mostly positive findings and offer implications for further strengthening LDC. 


Overall Findings 


Teachers and administrators appreciated LDC and perceived positive impact on their 
practice and their students’ learning. Quasi-experimental analyses tended to produce positive 
estimates for the impact of LDC on student English language arts (ELA) assessment scores, but 
the differences did not reach the level of statistical significance. Specifically, 


Participants across all groups perceived a positive impact on student outcomes. A 
large majority of both teachers and administrators agreed that LDC helped improve 
student learning across multiple areas, including college and career readiness, literacy 
performance, writing, and content knowledge. 


Across all 3 years of the study, teacher, teacher leader, and administrator participants 
reported very positive attitudes toward LDC. Teachers appreciated the opportunity to 
collaborate and share practice, generally reported that the range of in person and 
digital supports was helpful, and agreed that participating in LDC positively impacted 
their practice, in areas such as engaging students in complex text, locating evidence of 
standards in final student work, and engaging students in understanding the 
performance task assignment and standards-rubric. Administrators generally saw LDC 
as a productive tool for meeting school instructional goals, but had some concerns 
with the intensive weekly time commitment needed to implement the program, with 
some administrators choosing to discontinue participation or dedicate less time to the 
program in favor of other priorities. 


While the LDC effect estimates tended to be in the positive direction, none of the 
quasi-experimental analyses yielded statistically significant results. Higher than 
expected levels of school and teacher attrition, and several changes in LDC project 
leadership are likely to have contributed to this lack of statistical significance. The 


1 An LDC module is a standards-embedded performance task assignment that explicitly guides students to write in 
response to reading complex and discipline-specific texts. The LDC module includes an assignment prompt and 
accompanying backwards-design instructional plan for teachers to implement in the classroom. 
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study does not, therefore, provide evidence that students taught by teachers who 
participated in the LDC program performed at different levels from matched 
comparison students on the New York State ELA Assessment. 


Professional Learning Community and Teacher Collaboration 


Nearly all LDC teachers participated in LDC-oriented PLCs. The frequency with which 
PLCs met in 2018-2019, however, varied greatly across schools, with the number of 
recorded meetings ranging between six and 37 times in the year, and averaging 12.9 
times. The average teacher’s individual attendance rate was 74%, but again there was 
great variation. A little over half of teachers met the teacher-level fidelity threshold of 
80% attendance. Only one fifth of schools met the program goal of three quarters of 
PLC participants attending 80% of meetings, with many schools experiencing 
challenges related to protecting planning time and ensuring that teacher participants 
attended PLC meetings. Both teacher surveys and PLC reflection forms indicated that 
PLC meetings typically lasted 45 to 59 minutes or an hour or more, and therefore met 
the LDC expectation. 


Teachers valued the collaborative nature of LDC and its PLCs. A large majority of 
teachers credited LDC with making them more likely to collaborate with other 
teachers, not only within their grade levels and content areas but outside of them as 
well. 


LDC Training and Support 


Teachers were nearly uniform in their positive attitudes about the value of their PLC 
participation. They found the PLCs a safe space for sharing instructional plans, 
problem solving, and learning to develop modules. 


Teacher leaders were almost universally reported to be highly approachable, 
supportive, knowledgeable, and helpful. 


Teacher leaders reported high satisfaction regarding the support they received from 
coaches, PD offerings, and how the teacher leader role allowed them to be 
instructional leaders in their schools. 


Overall, LDC coaches received positive feedback on the survey, with 93% of teachers 
and 97% of teacher leaders reporting that their coaches gave them appropriate and 
timely feedback and support. Data suggest room for improvement when it comes to 
frequency and usefulness of coach feedback. While coaches were somewhat more 
likely to meet fidelity thresholds on module comments in 2018-2019 than 2017-2018, 
peer review was used even less frequently in 2018-2019 than it was used in 
2017-2018. 


Most teachers rated CoreTools positively, which demonstrated the success of changes 
LDC made prior to the 2017-2018 school year including refinement of the content, 
sequencing, delivery of CoreTools’ instructional content, and streamlining of 
participants’ learning process. Overall, teachers were enthusiastic about much of the 
online learning content, with three quarters or more of teachers rating most aspects 
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as good or excellent. About a quarter of teachers, however, were concerned with the 
ease of use of the online course materials. Despite teachers’ positive attitudes, 
analysis of CoreTools data revealed that they were being exposed to less online course 
content than program goals intended. There was some improvement in this area from 
2017-2018 to 2018-2019, with a few schools meeting fidelity of implementation on 
exposure to the instructional content, but still exposure rates were considerably lower 
than fidelity thresholds. It is possible that there was not sufficient PLC time to cover 
the content with which LDC intended participants to engage. 


LDC Implementation 


Teachers typically reported adapting/creating and implementing between two and 
three LDC modules across the school year, which meets LDC program expectations. 


Analysis of program data suggests that while nearly all participants were engaging 
with the module-building platform to some extent, the level of engagement varied 
greatly across individuals and across subgroups (role, cohort, school level, and content 
area) as evidenced by the wide range in the number of views, edits, and comments 
across teachers. More in-depth analysis of the portions of the modules that teachers 
edited online indicated that engagement varied greatly across teachers’ user 
accounts, with about half of user accounts associated with engagement at a basic level 
(editing the teaching task), and other accounts associated with deeper engagement 
(editing multiple portions of modules). It is important to note that in response to 
teachers collaborating on and implementing common modules in schools, LDC refined 
its data collection in the summer of 2018 to include tracking engagement data on 
modules that teachers collaborated on. The fidelity matrix, and CRESST’s analyses of 
editing data, were not designed to capture this shift toward a collaborative model of 
instructional design in CoreTools, and therefore may not fully capture engagement in 
the design process. 


The majority of teachers (81% to 92%) reported success in nine key areas of LDC 
module development. Teachers were most confident in selecting focus standards, 
creating the writing assignment, and identifying skills needed in the module. The 
module analysis, however, suggests that the materials adapted and created by PLC 
members varied in levels of completion and quality. 


With regard to their classroom implementation of LDC modules, the majority of 
teachers reported success with all six key areas queried (84% to 91%). Teachers were 
most confident with engaging students in complex text, locating evidence of standards 
in final student work, engaging students in understanding the assignment and rubric, 
and providing feedback to students using the assignment rubric. 


Leadership Support 


Almost all teachers and teacher leaders reported that their administrators encouraged 
LDC participation at the school. The majority of teachers and teacher leaders agreed 
with administrators that they allocated resources to ensure that LDC teachers could 
participate in meetings. Administrators generally voiced strong support for LDC, but 


varied in their level of direct engagement with the work, with some taking a hands-on 
approach by attending many PLC meetings, and some not. 


Both administrators and teacher leaders were less likely to take full advantage of the 
in-person meetings offered by LDC in 2018-2019 than they were in 2017-2018, with 
administrator engagement particularly low. On the other hand, coach/teacher leader 
calls were more consistent in 2018-2019 than in 2017-2018; nevertheless, many 
schools did not meet program goals related to the frequency of calls. 


Impact on Teacher Practice 


The majority of teachers reported improving their practice in seven LDC-related skills 
(79% to 85%). Teachers were a bit more likely to report impact on selecting focus 
standards for an assignment, creating standards-driven writing assignments, and 
identifying skills that students need in writing assignments (skills concentrated at the 
beginning of the LDC learning cycle). 


Over 80% of teachers agreed that participating in LDC raised their expectations for 
students’ writing, helped them incorporate writing assignments into their existing 
curriculum, and made them more likely to collaborate with other teachers on 
designing instruction. 


Among teachers who completed the survey in both 2017-2018 and 2018-2019, 
attitudes around impact on teacher skills and practices were on average even more 
positive in 2018-2019 than in 2017-2018. 


Overall, CRESST’s ratings of module quality decreased from 2017-2018 to 2018-2019, 
when looking at all rated modules. The results for those modules created by a 
subpopulation of teachers who were present in both years were more mixed. Please 
note that sample sizes were small and thus comparisons across years were 
exploratory in nature. 


Sustainability 


Survey respondents generally reported confidence that the LDC program would be 
sustained in their schools. Analysis of attrition patterns, however, revealed concerns 
about sustaining the LDC initiative in schools. Even with the substantial supports 
provided by LDC, a number of administrators decided over the course of the 3 years 
that they did not have the resources (particularly staff time) to remain in the program. 
Competition for time between different reform efforts and school priorities seemed to 
play a role in whether schools remained in the LDC program. 


Impact on Student Learning 


Survey respondents were nearly uniform in perceiving a range of positive impacts of 
LDC on student learning. Among teachers who completed the survey in both 
2017-2018 and 2018-2019, attitudes about LDC impact on students were on average 
even higher in 2018-2019 than in 2017-2018. 


e Although effect estimates tended to be in the positive direction, the study provides no 
evidence of a statistically significant difference in ELA assessment scores between 
students taught by LDC teachers and matched comparison students. 


Conclusions 


Implementation data demonstrated great appreciation for LDC among teachers, teacher 
leaders, and administrators. Educators found LDC to be a helpful tool for fostering 
collaboration, creating a safe space for sharing practice, and increasing teacher skills and 
knowledge around literacy instructional design and teaching. Despite broad support, however, 
many schools and teachers did not meet fidelity thresholds related to attendance, exposure to 
instructional content, and principal observations of LDC instruction. The findings demonstrate 
the importance of school leaders and teachers understanding and committing to the program 
in advance of implementation, and school leaders dedicating substantial resources to the 
program, particularly time for common planning. 


Quasi-experimental analyses did not provide evidence of a statistically significant impact 
of LDC on student learning, as measured by New York State ELA Assessment scores. Effect 
estimates, however, tended to be in the positive direction. The program was challenged by 
substantial attrition in both cohorts of schools and teachers, and fidelity of implementation 
varied greatly across schools. Improved retention of teachers and higher fidelity could 
potentially lead to greater effectiveness in future LDC implementations. 


xi 


Literacy Design Collaborative 2018-2019 
Evaluation Report for New York City Department 
of Education 


Jia Wang, Joan L. Herman, Scott Epstein, Seth Leon, Deborah La Torre, and Velette Bozeman 


CRESST/University of California, Los Angeles 


1.0 Introduction 


The Literacy Design Collaborative (LDC) was created in 2009 to support teachers in 
implementing Common Core State Standards (CCSS) by integrating literacy skill development 
throughout the content areas. LDC is a national community of educators providing a teacher- 
designed and research-based framework, online tools, and resources for creating both literacy- 
rich assignments and courses across content areas. Used by individual teachers, schools, and 
districts in 40 states for the past 4 years, LDC also is a statewide-adopted strategy for CCSS 
implementation in Kentucky, Colorado, Louisiana, and Georgia. 


UCLA’s National Center for Research on Evaluation, Standards, and Student Testing 
(CRESST), in collaboration with its partner Research for Action (RFA), engaged in the evaluation 
of LDC implementation and its impact on student learning and teacher effectiveness starting in 
June 2011, via two parallel research studies funded by the Bill and Melinda Gates Foundation. 
Those studies included an examination of LDC implementation in eighth grade social studies 
and science classrooms in Kentucky and Pennsylvania and a districtwide implementation in 
sixth grade advanced reading classes in a large district in Florida. Results for the studies are 
available in two CRESST reports (Herman et al., 2015a; Herman et al., 2015b), as well as a 
journal article published by AERA Open (Herman et al., 2016). 


Because of its earlier success in helping students and teachers in middle schools, LDC 
received an Investing in Innovation (i3) validation grant to further develop the program. The 
current i3 project focuses on developing teacher competencies through job-embedded 
professional development and the use of professional learning communities (PLCs). Teachers 
work collaboratively with coaches to develop their expertise further and design standards- 
driven, literacy-rich writing assignments within their existing curriculum across all content 
areas. Currently, CRESST serves as the independent evaluator for LDC’s federally funded i3 
validation grant. 


While the initial focus of the i3 grant was to serve teachers and students in middle 
schools, LDC responded to local conditions by expanding its implementation to also include 
elementary and high schools at both study districts: New York City Department of Education 
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(NYCDOE) and a large urban school district on the West Coast. The comprehensive mixed- 
methods evaluation includes quasi-experimental design (QED) analyses designed to estimate 
the impact of LDC on student learning and document LDC’s impact on teacher skills and 
practices. Specifically, the evaluation study addresses a wide range of questions about program 
characteristics, conditions, and impacts in the context of the two school districts. The study 
measures teacher implementation and skill improvement with teacher surveys, analytic data 
from LDC’s online CoreTools module building platform, PLC reflection forms, and artifact 
analysis. While we document the core strategies of the LDC model as implemented and provide 
support for LDC improvement, the central focus of our comprehensive mixed-methods 
evaluation is to examine the impact of LDC on student learning using a QED. 


This report examines LDC implementation at NYCDOE schools during the 2018-2019 
school year, and the impact of LDC on student learning across 2017-2018 and 2018-2019. It 
draws on data from two cohorts of schools, with each school housing a PLC of teachers who 
engage in professional learning about LDC and implement LDC mini-tasks and modules in their 
classrooms. The first cohort of schools began implementing LDC during the 2016-2017 school 
year, and the second cohort began implementation during the 2017-2018 school year. A 
parallel report focusing on implementation in the West Coast school district was prepared and 
shared in February 2020. 


In this report, we present results from (a) surveys of classroom teachers, LDC teacher 
leaders, and school administrators; (b) analyses describing how LDC participants interacted with 
the CoreTools module building platform; (c) CRESST ratings of instructional modules created by 
LDC participants; (d) analysis of the fidelity of implementation across multiple key components, 
indicators, and data sources; and (e) primary and supplementary student outcome analyses 
using QED techniques. 


Survey results provide a window into how LDC was implemented in 2018-2019 (the final 
study year), the perceived utility and effectiveness of various program components, and the 
perceived impact of LDC on both teacher and student skills and knowledge. The analysis of 
CoreTools user data and modules created and adapted by teachers provides evidence on the 
level of engagement with the online platform and module design process, and the quality of the 
products created by the teachers and PLCs. Fidelity of implementation analysis, utilizing the 
fidelity matrix designed collaboratively by LDC and CRESST, provides a broad picture of how 
schools and the program as a whole performed on key fidelity indicators in 2018-2019. 


Our primary quasi-experimental analyses illuminate the impact of LDC on students, as 
practiced by teachers who amassed 2 or more years of experience with LDC. In addition, 
supplementary analyses examine the impact of Cohort 2 teachers alone and the impact on 
subgroups of students based on whether they were exposed to LDC in the prior year and how 
much exposure they had to LDC in the outcome year. 
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1.1 Logic Model 


The logic model includes four key intervention components that are predicted to be the 
drivers of change in teacher practice and student learning (see Figure 1.1). These components 
include a coach-supported “professional learning community” formed to implement the LDC 
intervention at the school site and provide a space for teacher collaboration; “asynchronous 
support from coaches” primarily in the form of feedback in CoreTools through comments and 
peer review; “implementation activities” completed by participating teachers including module 
development and classroom implementation; and “leadership support” at different levels. Note 
that the model also indicates LDC’s expectations for the level of implementation in each area. 


Figure 1.1 
LDC i3 Logic Model 


LITERACY DESIGN COLLABORATIVE i3 LOGIC MODEL 


Activities Participation Short term Medium term Long term 


Common Plannin Student Increased Students 
time for LDC o - PLC members participate in at least 60 engagement CCRS skills acquire 
F minutes of common planning time during knowledge and 
Professional . 
Learning each consecutive two weeks, 
Digital coach joins PLC every other week. 


Community (PLC) hae 
with synchronous i ae blocks from LDC 


coach support 


Improved skills to be 
test scores college and 
career ready 
Higher rates 

of course Educational 
completion attainment 
increases 


Increased teacher 
expertise and skill 
development 


Asynchronous 
support from LDC 


Coach provides feedback on modules in 
CoreTools via comments and peer review 
rubric ratings. 


Higher 
graduation 
rates 


coaches 


Increased labor 

market 

Teacher - Teachers develop at least one module per outcomes 

implementation year in Years 1-2. 

activities - Teachers teach at least two modules per year 
in Years 1-3. 


More effective 
CCRS-aligned 


instruction 
District instructional specialists visit schools and use of 
three times per year to sit inon PLC, meet with formative 
Leadership principal and teacher leader, and observe assessment 
support at instruction. 
different levels - Principals participate in LDC by protecting PLC 


time, setting goals for LDC work, facilitating & 
participating in school visits from district 
instructional specialists, and observing LDC 
instruction. 
Teacher leaders facilitate PLC on the ground, 
collaborate with administrator, and 
communicate regularly with coach. 


Contextual Factors: District support, school and student demographics, other policies and initiatives 


The logic model predicts that the four key components will lead to increased teacher 
expertise and skill development and more effective CCSS aligned instruction that incorporates 
ongoing formative assessment. In turn, increased teacher capacity and more effective 
instruction will lead to increased student engagement in the short term; increased student skill 
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acquisition, higher test scores, and higher rates of course completion in the medium term; and 
improved college and career readiness, education attainment, graduation rates, and labor 
market outcomes in the long term. 


Note that the logic model was revised based on refinements to the program in response 
to learning from both the pilot year (2015-2016) and Cohort 1’s first year of implementation in 
2016-2017. 


1.2 Evaluation Questions 


Our evaluation questions focus on addressing three main areas: program characteristics 
and implementation, contextual factors and implementation, and program impacts. This is the 
first report to provide results on our primary outcomes analyses, estimating the impact of LDC 
on student learning, as implemented by teachers participating in the program for two 
consecutive years. 


1. Program Characteristics and Implementation 


a. Who are the participating teachers and schools? Are they representative of the 
teacher/school populations of the respective district on years of teaching, 
education level, prior student performance, etc.? 


b. How is the LDC program implemented in each district? What are the core 
components (e.g., training, tools, on-site or other direct support) and who are the 
key participants? In what ways did the LDC implementation align with the 
intended model? 


c. In what ways do teachers implement the LDC tools in their classrooms? To what 
extent do teacher practices align with intended LDC practices? 


d. How are teachers utilizing the online LDC system (including online tools, 
exemplars, collaborative work spaces, and technical assistance) in terms of 
frequency and use of key features? Does this vary by teacher characteristics? What 
are teachers’ perceptions of the value and quality of the online LDC system? 


e. What types of LDC professional development opportunities are offered to and 
utilized by teachers at each school/district? Are teachers and schools satisfied with 
the LDC professional development opportunities they received? 


2. Contextual Factors and Implementation 


a. What factors facilitate or hinder successful implementation of the LDC model at 
the teacher, school, and district levels? 


b. How can implementation of the model be improved at the teacher, school, and 
district levels? 


c. What other educational reforms are being implemented in the participating 
schools and districts? What are their influences on the LDC adoption in the schools 
and districts? Are schools able to align reform efforts? 
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d. 


What are the roles of school and district leadership in shaping the LDC 
implementation? 


3. Program Impacts 


a. 


What is the impact of LDC on the academic performance of participating students 
as measured by the state assessments? 


Do the academic impacts vary by student subgroup including prior achievement, 
race, ethnicity, socioeconomic status, gender, language proficiency, and/or 
disability? Does LDC help close the achievement gap between student subgroups? 


Do the academic impacts vary by student grade level or subject? 


What is the impact of LDC on teacher skill improvement and learning as measured 
by CoreTools and by the quality of LDC modules they produce? What is the self- 
reported impact of LDC on teacher learning? 


To what extent do teachers report changes in their practice (e.g., teaching 
strategy, collaboration with others) and changes in their comfort in implementing 
CCSS during and after the LDC intervention? 


What is the relationship between the fidelity of implementation, fidelity of 
intervention, and student learning? What are the conditions and contexts under 
which LDC tool use is most effective? 


To what extent do Cohort 1 participating schools and teachers continue their LDC- 
influenced practices in the 2019-2020 school year after the LDC support ends? 
What contributed to their decision to continue or stop? What factors contributed 
to their levels of continued implementation? How does Cohort 1’s actions align 
with their previously stated intentions for continuation of LDC-influenced practices 
as reported in spring 2017? To what extent do Cohort 2 participating schools and 
teachers plan to continue their LDC-influenced practices after the LDC support 
ends? 
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2.0 Study Methodology 


In this chapter, we provide an overview of the methodology behind this report on LDC 
implementation and impact. We begin by describing the various instruments and data sources 
for the analyses, including surveys of classroom teachers and teacher leaders participating in 
PLCs as well as administrators overseeing the implementation; analytic data from LDC’s 
CoreTools platform; module artifacts including samples of student work; LDC administrative 
data; and district administrative data on students and teachers used for the outcome analyses. 
We then describe the sample of educators and schools for each of these data sources. Finally, 
we discuss the methodological approaches for the various analyses we conducted. 


2.1 Data and Instruments 


We next describe each of the data instruments and the elements they contain. Most 
variables are measured at the teacher level, which is the unit at which the LDC intervention is 
being implemented. Administrative data for the analysis of the impact of LDC on student 
learning includes school-, teacher-, and student-level variables. 


Surveys (Teachers, Teacher Leaders, and Administrators) 


This report includes findings from the third and final year of survey collection in 
2018-2019. In collaboration with LDC, CRESST previously made minor revisions to the 
2016-2017 surveys for the 2017-2018 survey administration. These refinements involved 
streamlining the language of certain questions, in particular those capturing teacher 
pedagogical practice and the perceived impacts of LDC on teachers and students. In a few 
cases, questions and items were also added to collect systematic information on program 
conditions and impacts that were observed by LDC and CRESST anecdotally. The 2018-2019 
surveys were substantively identical to the 2017-2018 surveys. 


In 2018-2019, four different surveys were administered to LDC participants playing three 
different roles: teacher, teacher leader, and administrator. Some teacher leaders were 
classroom teachers who implemented LDC with their students, while other teacher leaders 
were out-of-classroom faculty (coaches or coordinators); these two groups received different 
versions of the survey tailored to their roles. Thus, four versions of the surveys were 
administered in spring 2019: (a) teacher, (b) teacher leader (for coaches and coordinators), 

(c) teacher leader (for teachers), and (d) administrator. 


The surveys were designed to capture multiple perspectives on key aspects of LDC’s logic 
model (see Figure 1.1), and to provide data to answer the evaluation’s research questions 
presented earlier. Survey questions targeted at the three roles fall under the domains and 
subdomains in Table 2.1. Domains were selected to align with the LDC i3 logic model and with 
the CRESST evaluation questions. Note that most domains cover multiple subdomains, 
constructs, and survey questions. “Professional Learning Community/Teacher Collaboration,” 
for example, captures the intensity, frequency, and collaborative environment of common 
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planning time; “LDC Training and Support” includes quality of online courses, utility and 
effectiveness of coach support, etc.; and “LDC Implementation” encompasses module creation, 
classroom implementation of modules, and module peer review. 


Table 2.1 
Survey Domains for Three Respondent Groups 


Domain Teacher Teacher Administrator 
Leader 

LDC Participation X X X 
Professional Learning Community and Teacher Collaboration X X X 
LDC Training and Support X X X 
LDC Implementation 

Module Creation X X 

Classroom Implementation X 

Module Peer Review X 

Alignment X X 
Leadership Support 

Teacher Leader Support X 

School Administrator Support / Classroom Observation X X X 

Teacher Leader Leadership Role X X X 

District Support X X 
Impact 

Impact on Teacher Practice and Learning X X 

Impact on Student Learning X X 
Scale-Up and Sustainability X X 
Facilitators and Barriers X 
Areas of Improvement X X X 


Teachers and administrators were asked to reflect on both LDC’s “Impact on Teacher 
Practice and Learning” and “Impact on Student Learning.” Questions within a number of 
domains further asked respondents to reflect on conditions and supports that may potentially 
impact LDC’s implementation. These domains included teachers’ perceptions of “Facilitators 
and Barriers” to implementation and perceptions regarding leadership roles and support for 
LDC at different levels. Teacher leaders and administrators were also asked for their 
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perceptions regarding if and how LDC would be sustained and expanded within the school. 
Finally, all respondents were asked open-ended questions regarding “Areas of Improvement” 
for LDC implementation. Teacher, teacher leader, and administrator survey results can be 
found in Chapter 3, and the survey instruments can be found in Appendices A, B, and C. 


LDC CoreTools 


The CRESST team received the LDC program data on i3 participants’ interactions with the 
CoreTools module-building platform. The data files captured a number of behaviors including 
document page viewing, document editing, document commenting, coach use of peer review, 
exposure to LEARN instructional cycle content, and uploading of student work. The data 
contained date- and time-stamped records of participants’ activities in all of these areas. 


Using these data files, we were able to analyze variation in the number of times the 
participants viewed, edited, and commented on documents across the school year. We 
generated descriptive statistics (minimum, maximum, mean, and standard deviation) for the 
number of times participants viewed a document page, edited a module document, and 
commented on a module document. We then produced descriptive statistics on these 
behaviors for each role group (teacher, teacher leader, and administrator), school level 
(elementary, K-8, middle, 6-12, and high schools), cohort, and content area subgroup. We also 
compared the average engagement in these key activities across two groups: teachers who 
implemented modules as measured by the presence of uploaded student work and those 
teachers for whom we do not have evidence of module implementation. The results on these 
overall metrics of engagement are reported in Chapter 4. 


CoreTools data were also used to analyze to what extent (a) teachers were exposed to 
instructional cycles of LEARN content; (b) coaches provided feedback via commenting and peer 
review functions; (c) teachers edited key portions of modules; and (d) teachers uploaded 
student work (a proxy for classroom implementation). Results for these indicators are reported 
in Chapter 6. 


Modules 


We adapted the existing module rating rubrics (Reisman et al., 2013) that were developed 
for earlier studies of LDC. The rubrics were designed to examine the instructional quality and 
coherence of the LDC modules, and to address the rigor in both content and literacy 
development materials (i.e., template task, student work samples, and descriptions of the 
pacing and goals of the modules). The first two dimensions examined the quality of the 
teaching task, while the remaining four dimensions focused more holistically on module quality: 
(a) effective writing task; (b) alignment to the CCSS and local and state literacy and content 
standards; (c) fidelity to LDC module instruction; (d) quality instructional strategies; (e) 
coherence and clarity of module; and (f) overall impression. 
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Each of the dimensions was rated using a 5-point scale with anchor points on the first five 
dimensions ranging from “not present or realized” to “fully present or realized” and the final 
dimension ranging from “inadequate” to “advanced” LDC module implementation. This scale 
mimics the structure of the three-level LDC jurying system scale that uses the ordered anchors 
of “work in progress,” “good to go,” and “exemplary quality.” For each dimension, a 1 
represented the lowest possible level of quality, while a 5 represented the highest level of 
quality (see Appendix D). 


LDC Administrative Records 


The fidelity matrix analyses in Chapter 6 utilize administrative records collected by LDC as 
part of their ongoing program management. These records include (a) PLC reflection forms, and 
(b) attendance records for in-person meetings organized for the benefit of school 
administrators and teacher leaders. The PLC reflection form captures attendance at regular PLC 
meetings, the amount of time spent on LDC during these meetings, whether teacher leaders 
had a separate planning call with coaches, and the subjects that PLCs were working on during 
sessions. 


Administrative Data Used in Student Outcomes Analysis 


Student-level variables utilized in the outcome analysis included race/ethnicity, gender, 
poverty status, special education status, English language proficiency, grade, and baseline 
achievement in mathematics and English Language Arts (ELA), as well as outcome year 
achievement in ELA on state assessments. Teacher level indicators obtained and utilized 
included years of teaching experience and teacher attendance. We also requested and received 
roster files that establish a link between teachers and students via specific courses. 


2.2 Study Population and Generalizability 


Here we describe the population of schools and teachers who participated in LDC, how 
that population changed over time, and how that population compared to all schools and 
teachers in the district. Over the course of the project, two cohorts of schools were recruited to 
participate in LDC, with one cohort beginning implementation in 2016-2017 and a second 
cohort beginning in 2017-2018. The initial 29 Cohort 1 schools included two elementary 
schools, 17 middle schools, three high schools, four K-8 schools, and three 6-12 schools. 
Cohort 2 included 16 elementary schools, 11 middle schools, five K-8 schools, and three 6-12 
schools. As displayed in Table 2.2, there was substantial attrition at both the school and student 
levels for each cohort. Over one third of Cohort 1 schools dropped out of the program after 
2016-2017, and within the remaining Cohort 1 schools, 53% of teachers did not continue with 
LDC in 2017-2018. Attrition in Cohort 2 was somewhat less pronounced but still very 
substantial. About one third of schools did not continue with LDC in 2018-2019, and within the 
remaining Cohort 2 schools, 36% did not continue with the program in 2018-2019. 
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Table 2.2 
Attrition of Cohort 1 and Cohort 2 Teachers, Administrators, and Schools 


Sample Teachers Administrators Schools 


Cohort 1 (beginning 2016-2017) 


Participated in LDC in 2016-2017 219 47 29 
Remained in LDC in 2017-2018 52 22 18 
Attrition rate as of 2017-2018 76% 53% 38% 
Remained in LDC in 2018-2019 28 16 13 
Attrition rate as of 2018-2019 87% 66% 55% 
Cohort 2 (beginning 2017-2018) 
Participated in LDC in 2017-2018 229 54 35 
Remained in LDC in 2018-2019 97 28 23 
Attrition rate as of 2018-2019 58% 48% 34% 


To better understand teacher and school attrition, we conducted an interview study of 
school administrators in the 2017-2018 school year. Twenty principals and assistant principals 
from both the West Coast district and our parallel New York City Department of Education 
study volunteered to be interviewed. The principal interview responses revealed a diversity of 
reasons. The main reason for teacher retention involved teacher and grade-level team 
decisions to leave or stay, followed by the principals’ decision to switch participation of 
teachers between the 2 years. Buy-in also seemed to affect teachers’ decisions on whether to 
continue. Although the sample was small, one of the reasons schools seemed to have dropped 
out was because they were overburdened by other priorities, including other reform efforts in 
the districts. Information submitted by school administrators to LDC confirmed that there were 
many other reform initiatives happening in schools during the study, including programs related 
to ELA, such as Early Language and Literacy Plans, Scholastic Leveled Bookrooms, and Achieve 
3000. See Appendix E for a detailed description of the results as well as the methodology for 
the interview study. 


In Table 2.3, we provide a snapshot of participation in 2018-2019, the final 
implementation year, by cohort, the year the teacher or administrator began participating in 
LDC, and school level. Over one third of participants were from elementary schools, a little less 
than a third from middle schools, and a little less than a third from either K-8 or 6-12 schools. 
Over two thirds of participants were from Cohort 2 schools. Both cohorts of schools also had 
teachers and administrators join LDC after the first year of implementation. Ninety-six teachers 
and administrators in Cohort 1 schools joined LDC in either 2017-2018 or 2018-2019, including 
18 who participated in both 2017-2018 and 2018-2019, 49 who participated only in 
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2017-2018, and 29 who participated only in 2018-2019. These participants are not part of our 
primary impact analyses; however, we conducted supplementary analyses to evaluate their 
impact on student learning when there was sufficient sample. 


Table 2.3 
Number of Participants in 2018-2019 by Cohort/Start Year and School Level 


Cohort/Start Year Elementary K-8 Middle 6-12 Total 
Cohort 1 school/started 2016-2017 8 10 17 9 44 
Cohort 1 school/started 2017-2018 2 1 13 2 18 
Cohort 1 school/started 2018-2019 2 6 12 9 29 
Cohort 2 school/started 2017-2018 55 27 31 12 125 
Cohort 2 school/started 2018-2019 47 16 21 1 85 
Total 114 60 94 33 301 


Next, we describe how the participating schools, teachers, and students in the LDC 
program compared to those in the school district as a whole. In Table 2.4, we provide a 
snapshot for each of the three study years. For each year, we present characteristics of schools 
and their grade 3-8 student populations compared to all schools and all grade 3-8 students in 
the district. We chose to focus our reporting of student characteristics on the grade 3-8 
population, because our quasi-experimental analyses are based on that grade range. We also, 
however, examined the characteristics of all students in grades K-12, and those results were 
very similar to those reported in Table 2.4. 


Not surprisingly given LDC’s focus on middle schools, the LDC treatment group had a 
greater proportion of middle schools and a smaller proportion of elementary schools than the 
district as a whole. The student population in LDC schools was consistently more likely to be 
Black or Hispanic, and less likely to be White or Asian/Pacific Islander than the district’s overall 
student population. Students in LDC schools were also more likely to be economically 
disadvantaged. Students were classified as Limited English Proficient and eligible for special 
education services in similar proportions in the two groups. LDC schools were substantially 
lower performing in both ELA and mathematics than all schools in the district serving students 
in grades 3-8. 
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Table 2.4 


Characteristics of LDC Schools and Grade 3-8 Student Population at LDC Schools Compared to Students 
in New York City Department of Education as a Whole, by Year 


2016-2017 2017-2018 2018-2019 
vr ee Oe ee ae 
School level 
Elementary schools (%) 8 51 35 51 33 53 
Elementary/middle schools: K-8 (%) 19 14 15 14 22 12 
Middle schools (%) 58 20 40 20 33 21 
Middle/high schools: 6-12 (%) 15 7 10 7 11 8 
K-12 schools (%) 0 8 0 8 0 7 
Student level characteristics 
Hispanic (%) 51 Al 47 Al 47 41 
Black (%) 34 26 38 26 37 22 
Asian/Pacific Islander (%) 7 15 7 15 9 18 
White (%) 6 15 5 15 4 17 
Female (%) 51 49 A8 49 50 48 
Limited English proficient (%) 17 15 16 15 16 16 
Special education (%) 25 21 23 22 23 23 
Poverty (%)* 86 71 86 76 84 74 


Student Achievement 
Mean standardized ELA achievement -0.346 0.00 -0.320 0.00 -0.296 0.00 


Mean standardized mathematics -0.441 0.00 -0.393 0.00 -0.394 0.00 
achievement 


Next, we explore how the main LDC teacher groups of interest compared to teachers 
district wide (see Table 2.5). These groups are Cohort 1 teachers with 2 years of LDC 
participation as of 2017-2018 and Cohort 2 teachers with 2 years of LDC participation as of 
2018-2019. We report on a number of teacher characteristics, as well as the mean 
achievement of these teachers’ students in the baseline year. Not all variables were available 
for each of the two cohorts/years. For example, demographic characteristics were not available 
for 2018-2019. The analysis suggests that Cohort 1 teachers on average had less experience 
than their peers in the district as a whole, and Cohort 2 teachers on average had more 
experience. Cohort 1 and Cohort 2 teachers also taught students that were substantially lower 
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performing on average at baseline than grade 3-8 students in the district as a whole. Cohort 1 
teachers more likely to be Black and less likely to be White than in the district as a whole. 


Table 2.5 


Characteristics of LDC Core Content Area Teachers with Two Consecutive Years of LDC Participation and 
New York City Department of Education Teachers, by Cohort and Year 


Cohort 1 in 2017-2018 Cohort 2 in 2018-2019 


Variable LDC a LDC ela 
Hispanic teachers (%) 14 15 NA NA 
Black (not Hispanic) teachers (%) 29 19 NA NA 
Asian (not Hispanic) teachers (%) 7 6 NA NA 
White (not Hispanic) teachers (%) 46 58 NA NA 
Female teachers (%) 61 74 NA NA 
Teachers with regular/permanent assignment (%) 93 97 97 98 
Teacher mean years of experience 8.4 10.4 13.4 10.5 
a 
Mean standardized mathematics achievement at 0.43 0.01 0.46 0.00 


baseline of students taught by teachers 


2.3 Sample for Implementation and Outcomes Analyses 


Here we explore the teacher and administrator samples for the various implementation 
analyses in the final study year (2018-2019), as well as for outcomes analyses across the multi- 
year study. Table 2.6 displays the overall population and sample sizes for the different study 
measures and analyses included in this report. This includes all participants in 2018-2019. As 
can be seen, data were available for a majority of the participants. Seventy-eight percent of 
teachers consented to participate in the study, with 81% of all teachers completing the survey 
in spring 2019 (some teachers who did not return consent forms nevertheless completed the 
survey). Eighty-seven percent of administrators consented to participate in the study, and 45% 
completed the survey. Nearly all teachers (98%) and most administrators (82%) were present in 
the CoreTools dataset, which was provided directly to CRESST by LDC, and did not depend on 
teachers’ individual study consents. Most teachers (87%) were present in the PLC reflection 
attendance data as well (six schools did not properly record the data and therefore there is no 
attendance data on teachers at these schools). 
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Table 2.6 


Number of Participating Schools and Teachers in the District and Sample Size for Different Instruments, 
2018-2019 


Instruments Classroom teachers Administrators Schools 

Participated in LDC 254 47 36 
Consented to CRESST study 199 41 35 
Completed survey 206 21 35 
Present in CoreTools dataset 250 39 36 
Authored/coauthored a module with 153 0 35 
student work uploaded 

Present in PLC reflection form 591 36 30 
attendance records 

Leader in-person meeting attendance NA NA 29 
Included in outcomes analysis for 69 NA 33 


2018-2019 


The school district required individually signed consent forms before releasing teacher 
data and teacher/student rosters, so for the outcome analysis, we only received data on 
teachers who consented to participate in the study. Samples were further restricted by the 
need for student achievement data for both the outcome year (2017-2018 or 2018-2019 for 
Cohorts 1 and 2 respectively) and the baseline year (2015-2016 or 2016-2017 for Cohorts 1 
and 2 respectively). As a result, participants teaching either in high school or the primary 
elementary grades (K-3) were not included in any of the student outcome analyses. Middle 
school teachers who did not teach a core ELA, science, or social studies/history class were also 
excluded from the analyses. Of the 254 teachers from 36 schools who participated in LDC in 
2018-2019, 69 teachers from 23 schools were represented in at least one of the quasi- 
experimental analyses reported on in Chapter 7 with 2018-2019 as an outcome year. 


In Tables 2.7 and 2.8, we list all the potential teacher samples by cohort, school level, start 
year, and outcome year. Our previous reports examined the impact of LDC after 1 year for 
Cohort 1 and Cohort 2 elementary and middle school teachers. We also previously estimated 
the impact of Cohort 1 middle school teachers after 2 years of implementation. This final 
annual report includes models pooling across cohorts for each school level separately, and for 
the elementary and middle schools combined. We also examine the impact of LDC as 
implemented in Cohort 1 schools by teachers with 3 years of LDC participation. We also 
conducted supplementary analyses examining the impact of middle school teachers who joined 
existing PLCs in their schools’ second year of LDC implementation. 
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Table 2.7 


Number of Classroom Teachers by Cohort, Implementation Years, School Level, and QED Eligible Students 


(Cohort 1) 
Consented 
School level/ Outcome Years Consented teachers Summary of 
; Teachers : oa : 
start year year in LDC teachers with eligible reporting 
students 
Elementary school 
2016-2017. 2016-2017 1 26 25 14 gite auciz eee 
reports 
2016-2017 2017-2018 2 10 10 2 2020 report* 
2016-2017 2018-2019 3 6 6 0 Not conducted 
2017-2018 2017-2018 1 10 8 2 Not conducted 
2017-2018 2018-2019 2 2 2 0 Not conducted 
2018-2019 2018-2019 1 3 2 0 Not conducted 
Middle school 
2016-2017 2016-2017 1 147 133 104 ete and 2020 
reports 
2016-2017 2017-2018 2 35 33 26 ete anaien20 
reports 
2016-2017 2018-2019 3 17 17 11 2020 report 
2017-2018 2017-2018 1 51 31 17 eer 2026 
reports 
2017-2018 2018-2019 2 15 12 8 2020 report 
2018-2019 2018-2019 18 10 9 Not conducted 
High school 
2016-2017 2016-2017 1 57 51 0 NA 
2016-2017 2017-2018 2 7 7 0 NA 
2016-2017 2018-2019 3 3 3 0 NA 
2017-2018 2017-2018 1 10 6 0 NA 
2017-2018 2018-2019 2 2 1 0 NA 
2018-2019 2018-2019 1 7 4 0 NA 


Note. *There was insufficient sample size to test the impact of Cohort 1 elementary teachers in their 
second year of implementation independently, but these teachers were included in pooled analyses 
examining the impact of Cohort 1 and 2 elementary teachers after 2 years of implementation. 
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Table 2.8 


Number of Classroom Teachers by Cohort, Implementation Years, School Level, and QED Eligible Students 


(Cohort 2) 
Consented 
School level/ Years Consented teachers Summary of 
Outcome year Teachers : ate : 
start year n LDC teachers with eligible reporting 
students 
Elementary school 
2017-2018 2017-2018 1 137 129 54 2019 and 2020 
reports 
2017-2018 2018-2019 2 78 77 31 2020 report 
2018-2019 2018-2019 1 63 57 16 Not conducted 
Middle school 
2017-2018 2017-2018 1 56 53 31 2019 and 2020 
reports 

2017-2018 2018-2019 2 38 36 32 2020 report 
2018-2019 2018-2019 1 12 9 8 2020 report 
High school 
2017-2018 2017-2018 1 6 5 0 NA 
2017-2018 2018-2019 2 6 5 0 NA 
2018-2019 2018-2019 1 3 3 0 NA 


In addition to the CoreTools analytic files, we also received module artifacts from LDC for 
an analysis of the quality of module design. We restricted our analysis to modules that were 
created during the 2018-2019 school year and included original uploaded student work 
samples, because these samples were required for module scoring. That restriction yielded a 
pool of 137 modules that were authored or coauthored by 153 teachers (60% of all 


participating teachers), across 35 schools. Because the number of modules created surpassed 


available resources for scoring, we sampled the last module created by each teacher wherein 
they served as the sole or lead author. Our intent was to represent teachers’ best LDC work, 

and presumably, modules created later in the school year would be more skilled than those 

created earlier. The final total sample was 109 modules. Table 2.9 provides further detail about 
the distribution of modules. We also conducted an exploratory analysis of the 13 elementary 
teachers and 30 secondary teachers who created a complete module both this year and during 
the previous school year. 


27 


Table 2.9 


Background Variables for the Primary Module Analysis 


Elementary Secondary Total 
Variables # % # % # % 
Cohort 
Cohort 1 (2016-2017) 4 8.3 14 23.0 18 16.5 
Cohort 1 (2017-2018) 1 2.1 14 23.0 15 13.8 
Cohort 2 (2018-2019) 1 2.1 9 14.8 10 9.2 
Cohort 2 (2017-2018) 23 47.9 17 27.9 40 36.7 
Cohort 2 (2018-2019) 19 39.6 10 16.4 29 26.6 
Subject 
ELA 24 50.0 31 50.8 55 50.4 
Science 11 22.9 10 16.4 21 19.3 
Social Studies 13 27.1 20 32.8 33 30.3 
Total 48 44.0 61 56.0 109 100.0 


2.4 Survey Recruitment and Administration 


As in previous years, CRESST and LDC worked together to conduct the consent process for 
teachers and administrators who newly joined LDC in 2018-2019, and were successful in 
consenting 58% of these new participants. As required by the UCLA Office of the Human 
Research Protection Program, all consent forms included language stating that the study was 
voluntary, that data would be protected, and that by signing the form, participants consented 
to be emailed an electronic survey in spring 2019 and to have CRESST request district data 
linking them and their students. 


Directly compensating educators for participation in research is not permitted in New 
York State. We therefore offered instead gift cards of $300 to the school to be used for school 
supplies to directly benefit students. The gift cards were given to the school if the school’s 
teacher leader and at least five teachers participating in the PLC completed the survey. 


Surveys were administered in spring 2019. CRESST coordinated with LDC staff and coaches 
to administer the online surveys during common planning sessions. Multiple email reminders 
were sent to participants who were not present at these sessions or who otherwise did not 
complete the survey. The teacher survey administration was closed at the end of the school 
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year in June 2019. The administrator survey window was left open through August 2019 when 
administrators returned from their summer break. 


2.5 Module Scoring Process 


LDC requirements specified that all teachers implement two modules over the course of 
the year, with the first spanning 1 week based on one text and the second spanning across 
multiple weeks and incorporating multiple texts. Modules could be developed as original units 
of work or could be adapted from existing modules within the LDC CoreTools Library. Modules 
could also be either developed or adapted in solo or collaboratively with other teachers within 
the PLC. LDC modules were collected from elementary and secondary teachers who 
participated in LDC during the 2018-2019 school year. In total, 109 modules created by 
NYCDOE teachers were rated with 45 or 41.3% being rated by two panelists. Further details 
about the modules can be found in Appendix F. 


Eight expert raters with experience teaching in the targeted grade spans and content 
areas were recruited from Los Angeles county schools. Four panels were convened with two or 
three experts rating each of the following sets of modules: elementary ELA and social science, 
elementary and secondary science, secondary ELA, and secondary social science. It should be 
noted that one rater served on both the secondary ELA and the secondary social science panels. 
In addition, three of the teachers had 2 years of prior experience working as raters on the LDC 
module analyses and three had 1 year of prior experience. 


Separate trainings lasting approximately 2 hours were conducted for each module scoring 
panel. All trainings were conducted by a member of the evaluation team who is an expert on 
the CCSS and the rating of student and teacher artifacts, and who had been conducting the 
training for the past three cycles of scoring. The training included an overview of the LDC goals, 
template task, the structure of the modules, and the CRESST rating dimensions. Once the 
training was complete, calibration was conducted by having teachers individually score and 
then discuss their ratings for one module in the content area on which they would be focusing. 
Scoring was then conducted on subsequent days with each module individually rated. Those 
modules rated by two expert teachers were then discussed with the goal, but not the 
requirement to reach consensus (see Carlson & McCaslin, 2003). All discussions were facilitated 
by the same evaluation team member who conducted the initial training. 


2.6 Analytical Approaches 


Both quantitative and qualitative analytic methodologies were applied to the data to 
answer the evaluation questions about how LDC was implemented, conditions affecting 
implementation, and program impacts. The following describes the approaches used to analyze 
each data set. 
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Surveys 


Survey responses were analyzed using descriptive statistics for multiple-choice items and 
qualitative coding for open-ended responses. As previously noted, surveys were administered 
to teachers, teacher leaders, and administrators. Some teacher leaders were classroom 
teachers while others were coaches or coordinators. Because of this, the samples were not 
mutually exclusive, with teacher leaders who taught in the classroom represented in both the 
teacher and teacher leader results. Generally, we report the number and percentage of 
respondents who selected different multiple-choice options. In a few cases, where responses 
are numerical rather than categorical, we present means rather than proportions. Descriptive 
statistics for all multiple-choice items are presented in full in Appendix A for teachers, Appendix 
B for teacher leaders, and Appendix C for administrators. 


LDC CoreTools 


The first stage of our analysis examined the proportion of all participants who created 
CoreTools user accounts, and engaged in three key behaviors: viewing, editing, and 
commenting on modules. We then analyzed each of the three key measures of participants’ 
interaction with the LDC online tools, and reported frequencies and/or descriptive statistics 
(e.g., minimum, maximum, mean, standard deviation) as measures of participants’ engagement 
with the online LDC system. In addition to reporting the overall results, we provide results by 
cohort, the content areas taught (ELA, social studies/history, and science), by participant role 
(teacher, teacher leader, and administrator), and school level (elementary, K-8, middle, 6-12, 
and high) whenever feasible. Finally, we explored the difference in CoreTools engagement 
between two groups of teachers: those who completed and taught modules and those who did 
not appear to complete the design and implementation process. 


Modules 


We used descriptive statistics (means, standard deviations, and percentages) to analyze 
overall and subgroup performance for each content area and school level on each of the six 
dimensions. Additionally, generalizability theory (G theory) was used to examine potential 
sources of error during the rating process to help determine the validity of the scores as well as 
the construct validity of the rubrics (see Shavelson & Webb, 1991). Finally, teacher comments 
during the debriefings were examined to determine other potential issues with the rubrics 
and/or rating process. 


Fidelity of Implementation Analysis 


To examine fidelity of implementation we conducted descriptive analyses of a variety of 
data sources, including a teacher survey, CoreTools data, PLC reflection data, and attendance 
records for administrator and teacher leader in-person meetings. As outlined in the fidelity 
matrix (see Appendix G), fidelity levels are set at the teacher, module, school, and program 
levels. The process for most indicators involves several levels of aggregation, with, for example, 
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module editing being measured at the teacher level, school level, and program level. School- 
and program-level scores are typically based on the proportion of teachers or modules that met 
the threshold for adequate implementation. 


Table 2.10 


Summary of Fidelity Matrix Indicators and the Criteria for Meeting School- and Program-Level Fidelity 


Criterion for 
meeting fidelity at 
program level 


Criterion for meeting fidelity at 


Key component Indicator 
Y P school level 


Key Component 1: Teacher attendance at 75% of PLC teachers attended 80% of 75% of schools 


Common Planning weekly PLC meetings meetings met fidelity 
Time for LDC Amount of time spent PLC sessions typically met for 60 75% of schools 
eae on LDCin PLC sessions minutes plus met fidelity 
earning 
Community with Exposure to LDC LEARN 75% of PLC teachers viewed 60% of 75% of schools 
Synchronous content during 1st sessions in 1st LEARN instructional met fidelity 
Coach Support instructional cycle cycle 
Exposure to LDC LEARN 75% of PLC teachers viewed 60% of 75% of schools 
content during 2nd sessions in 2nd LEARN instructional met fidelity 
instructional cycle cycle 
Perceived effectiveness Teacher survey respondents on 75% of schools 
of engagementin PLC average reported moderate met fidelity 
on teacher improvement in key competencies 
competencies 
Key Component 2: Coach comments on 75% of modules developed at the 50% of schools 
Asynchronous modules school and linked to LEARN courses met fidelity 
Support from LDC received at least two comments from 
Coaches LDC coaches 
Coach formative peer 75% of modules developed at school 50% of schools 
review on modules and linked to LEARN courses received met fidelity 


feedback from LDC coaches via peer 
review rubric 


Teacher perception of 75% of teacher survey respondents 50% of schools 
the helpfulness of reported coach written feedback was met fidelity 
coach written feedback at least moderately helpful 
on modules 
Key Component 3: Module editing 75% of PLC teachers edited task and 75% of schools 
Teacher either standards or text in at least 1 met fidelity 
Implementation module 
Activities Module 75% of PLC teachers uploaded 75% of schools 
implementation student work to at least 2 modules met fidelity 
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An analysis of the relationship between fidelity of implementation and outcomes was also 
conducted. This analysis utilized the analytic sample from our primary quasi-experimental 
investigation of the impact of LDC as practiced by teachers with 2 years of LDC experience. Two 
models produced school and teacher residuals respectively. These residuals were then used to 
divide the sample into high, medium, and low achieving schools and teachers. Descriptive 
analysis and one-way ANOVA were used to examine whether mean fidelity scores differed for 
the three different achievement groups. Additional information on methodology can be found 
in Chapter 6, Section 7. 


Tables 2.10 and 2.11 summarizes the matrix indicators, and the criteria for meeting 
school- and program-level fidelity. Depending on indicator, the lowest unit of analysis was 
either a teacher, module, or school. The fidelity analysis produced a school-level fidelity score 
for each of the indicators. For the Key Component 1, 3, and 4 indicators, program-level fidelity 
was met if 75% of schools met fidelity. For Key Component 2 indicators, program-level fidelity 
was met if 50% of schools met fidelity. 


Table 2.11 


Summary of Fidelity Matrix Indicators and the Criteria for Meeting School- and Program-Level Fidelity 


Criterion for 


Criterion for meeting fidelity at meeting fidelity at 
Key component Indicator school level program level 
Key Component Frequency of Teacher leaders had at least 9 75% of schools 
4: Leadership coach/teacher leader planning / progress calls with met fidelity 
Support at monthly meetings coaches in school year 
Different Levels Administrator attendance School administrator participated in 75% of schools 
at quarterly in-person at least 2 out of 3 in-person meetings met fidelity 
administrator meetings 
Teacher leader Teacher leader participated in at 75% of schools 
attendance at quarterly —_ least 2 out of 3 in-person meeting met fidelity 
in-person teacher leader 
meetings 
Principal mini-task 75% of teacher survey respondents 75% of schools 
observation reported having their LDC instruction met fidelity 


observed by a school administrator 


Student Outcomes: Matching and Intervention Exposure 


Here we describe (1) our approach to selecting matched schools and students for our 
quasi-experimental design analyses estimating the effect of LDC on the New York State ELA 
assessment scores, and (2) our approach for modeling students’ exposure to LDC and non-LDC 
instruction in core content classes. 


32 


For all our analyses, we used a two-step matching process to identify a reduced pool of 
comparison students and teachers at schools with similar characteristics to the schools in the 
intervention sample. To accomplish this, we first identified up to five of the most similar 
comparison schools for each intervention school based on a Euclidian distance measure, by 
using the nearest neighbor analysis option in SPSS 24.0 (see Fix & Hodges, 1951; Wang et al., 
2007). The variables used in this process were the percentage of students eligible for free or 
reduced-price lunch, the percentage of Black students, mean baseline student achievement in 
ELA, mean baseline student achievement in mathematics, the average attendance rate of 
teachers, the average years of teaching experience of teachers, and the school grade span 
where feasible. We generally used all five identified comparison schools to establish the 
potential matching pool, but for selected LDC schools we only used four of the initial five 
matches. Once the pool of comparison schools was identified, their students and teachers were 
also identified, and student-level matching was conducted so that the resulting sample would 
resemble the type of sample one would expect to obtain through random assignment. 


The student-level matching technique we employed was coarsened exact matching (CEM) 
(lacus et al., 2011). CEM is a flexible matching approach with many favorable properties and 
allows the researcher to specify the precise conditions under which students are matched. For 
categorical variables, such as race/ethnicity or poverty status, this can entail exact matching, 
while for continuous measures, such as baseline individual student achievement and aggregate 
class-level achievement, cut-points for matching can be specified. With this approach we were 
able to set precise cut-points on the most important baseline indicators, such as baseline 
academic achievement, to ensure that where possible every treatment student was matched 
with a suitable comparison. Student matching variables we used in CEM included Hispanic, 
Black, poverty status, female, English language proficiency (English language learner), special 
education status, mean baseline achievement in mathematics and ELA, and grade level. 


During matching we also included a few variables capturing information on the teachers 
and peers to which students were exposed. These variables included mean prior ELA 
achievement of the student’s peers in their core content classes, and the average years of 
teaching experience of student’s core content teachers. 


The typical structure of course taking at the middle school level involves students 
potentially being exposed to multiple teachers, with each responsible for a different core 
content class. Specifically, middle school students in the study had exposure opportunities 
across three content areas (ELA, social studies/history, and science) taught by intervention 
teachers. As a result, students were not necessarily nested under individual teachers, but 
instead were likely to have connections to multiple teachers in the available time period prior 
to each testing outcome (students at the elementary school level were also sometimes exposed 
to multiple teachers but to a lesser extent). Therefore, LDC effects were estimated using an 
extension of the standard multilevel modeling framework known as multiple membership 
multiple classification (MMMC) models (Browne et al., 2001). 
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These models can account for complex classification structures, such as the LDC context, 
in which students are nested within schools but are also members of multiple classes led by 
different teachers who may or may not be implementing LDC. MMMC has the flexibility to 
account for this type of complex nesting structure in which students are hierarchically nested 
under schools but may have one-to-many relationships with teachers. There are three 
classification levels in the models we employ: students, teachers, and schools. In the MMMC 
modeling approach, each observation at the lowest level represents one student. Figure 2.1 
shows how students can be exposed to teachers in different content areas, who may or may 
not be participating in LDC. Our MMMC modeling approach can account for this complicated 
structure. 


Figure 2.1 
Variation in Student Exposure to Teachers 


Teacher 1 (ELA) 


Teacher 2 (History) Teacher 3 (Science) 


LESTNS< | 


Student 1 Student 2 Student 3 Student 4 Student 5 Student 6 


In the MMMC modeling approach, each observation at the lowest level represents one 
student. The weight each teacher receives with respect to each student is dependent on the 
student’s exposure to their teachers in each of the three core content areas. The total student 
exposure adds to a unity (i.e., a possible total exposure of 1) across their courses in the three 
content areas in a given school year. While this general weighting approach applies to both 
elementary and middle schools, the course structure of the datasets required us to use 
different weighting procedures in elementary and middle school. 


In both the elementary and middle school datasets, students were linked to teachers 
through statewide course identifiers and accompanying beginning and ending dates specifying 
the time each student was enrolled in a given course under a specified teacher. Students could 
potentially have data records connected to multiple teachers covering varying time periods in 
the same course. For each of the three core content courses we then collapsed the links into a 
single measure of the days of potential content exposure preceding the assessment period. The 
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codes aligning with the three core content areas of interest for our study were based on the 
School Codes for Exchange of Data (SCED) code handbook. In elementary school, in the event 
that a student was exposed to more than one teacher, each content area was given equal 
weight in distributing teacher/student exposure. For example, if a student was enrolled for 
both ELA and social studies/history under one teacher, then that teacher was coded as .67 for 
having contributed to two thirds of the students’ core curriculum exposure. If the same student 
enrolled in science with a different teacher than the one who was linked to their course marks 
in ELA and social studies/history, then that science teacher would have been coded as .33 and 
all other teachers in the sample would have been coded as zero. This would then result in the 
student’s exposure adding to a unity (1). 


In middle school, students’ exposure to teachers at the course level in the three core 
content areas was coded in the same manner as in the elementary grades based on enrolled 
time preceding the assessment period. A difference in our middle school coding process was 
that we did not force each core content area into equal weighting. Instead each core content 
area exposure contributed to a core content area total sum which formed the basis from which 
the weights were proportioned. Most commonly a student had equivalent days of core 
instruction exposure in each the three content areas (often 215 days in each content area). In 
that scenario, if a student had exposure to three different teachers, then each teacher would 
contribute one third (.33) of the overall core curriculum exposure and all other teachers in the 
sample would be coded as zero. However, in addition to the typical core science course, extra 
core science courses were also included in the LDC analysis (for example a Grade 8 student 
taking biology), which made it possible then for a student to accumulate more units in science 
than in the other two content areas. The weighting in middle school was always distributed as a 
proportion of the total exposure days in the three content areas. Therefore, if a student 
accumulated 300 science days (across two courses), two hundred social studies days, and two 
hundred ELA days, the base number of instruction days would be 700 days. If, using that same 
scenario, the same teacher taught both the typical core and biology courses then that teacher 
would contribute three-sevenths (.43) of the overall core curriculum exposure with the social 
studies and science teachers contributing two-sevenths (.285) each, again resulting in the 
student’s exposure adding to a unity (1). Tabular versions of the above examples can be seen in 
Appendix H. 


For this study, we modeled the treatment intervention variable as a fixed effect at the 
student level in two different ways. The first dosage dependent approach takes into account 
the students’ level of exposure to the intervention teachers. In this approach, the treatment 
was structured as a continuous response variable, coded as zero for comparison students and 
coded as a positive value for treated students, albeit never exceeding one. The positive value 
assigned to treated students in the dosage-dependent approach was simply the sum of the 
intervention teacher weights linked to the treated student. The second approach was dosage 
independent and classified any student exposed to an intervention teacher via at least one 
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course as a treated individual. In this approach the treatment variable was dichotomous (coded 
as one for treated students and zero for comparison students). 


Student Outcomes: Analysis Approaches 


As with other multilevel models, MMMC accounts for the non-independence of 
observations within cluster by adjusting the inferences on parameter estimates for the 
correlations between responses in a cluster. This modeling approach, however, becomes 
computationally cumbersome using traditional frequentist estimation methods. As 
recommended by Browne and colleagues (2001), to address this issue we employed Bayesian 
methods using Markov chain Monte Carlo (MCMC) techniques. Multilevel models incorporate 
demographic and achievement variables used in the matching design as covariates, making the 
findings “double robust” in that characteristics can be controlled for in both matching and 
outcomes analysis stages. Student demographic and baseline achievement variables that were 
used in the matching process were also included as covariates in the MMMC model. The full 
specifications for both models can be found in Appendix H. 


In Chapter 7, we report on both the primary and supplementary analyses of the impact of 
LDC. Primary analyses focus on impacts after 2 years of teacher participation in LDC (for both 
Cohorts 1 and 2), and impacts after 3 years of teacher participation in LDC (for Cohort 1 only). 
Data from both cohorts was pooled for the primary analyses measuring impact after 2 years of 
implementation to increase statistical power and to produce more precise impact estimates. 
For the pooled analyses, the outcome year for Cohort 1 was spring 2018 and for Cohort 2 was 
spring 2019. To maximize the sample, we also conducted a primary analysis pooling schools, 
teachers, and students across both cohorts (Cohort 1 and Cohort 2) and school levels 
(elementary and middle). For this, we assumed that being exposed to LDC in all core content 
areas at the middle school level was equivalent to being exposed to a single teacher 
implementing LDC at the elementary level. 


The other primary analysis reported on in Chapter 7 examines the impact of LDC as 
implemented by Cohort 1 teachers with 3 years of LDC experience on student ELA scores in 
spring 2019. This analysis focuses exclusively on middle school students due to the lack of 
achievement scores for elementary school students in the 2015-2016 baseline year. We also 
report supplementary analyses examining (1) the impact of Cohort 2 teachers with 2 years of 
LDC experience (these groups of elementary and middle school teachers are sub-samples of the 
pooled group of teachers in our primary analyses); and (2) the impact of LDC on subgroups of 
students broken down by whether they were exposed to LDC in the prior year, and how much 
LDC instruction they were exposed to in the outcome year. Where sample allowed, we also 
conducted analyses examining the impact of teachers who joined existing LDC PLCs in later 
years. 
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3.0 Survey Analysis 


In this chapter we present the survey results. First, we summarize teachers’ responses. 
We then summarize the teacher leaders’ responses, followed by the administrators’ responses. 
Within each of these sections, we organize results by the following domains as specified in the 
logic model: LDC participation; professional learning community and teacher collaboration; LDC 
training and support; LDC module implementation; leadership support; impact; and issues of 
scale-up and sustainability facilitators and barriers. After this we summarize the open-ended 
responses regarding supports for implementation of LDC. Then we present an exploratory 
longitudinal analysis of perceived impacts of LDC by teachers who submitted surveys in both 
2018 and 2019. We end with a summary of results. 


We use acronyms to identify which participants answered specific questions for each 
domain. We preface teacher items with “T,” teacher leader with “TL,” and administrator 
questions with “A.” For example, LDC Participation (T1—4) indicates that teacher survey items 
1-4 are used to provide information on LDC participation. Survey questions and descriptive 
results are presented in full in Appendix A for teachers, Appendix B for teacher leaders, and 
Appendix C for administrators. 


3.1 Teacher Survey Results 


As noted earlier, 206 teachers spanning 35 schools completed the survey. Among the 
teachers who completed a survey, the largest proportion taught an elementary school grade 
while the smallest proportion taught at the 6-12 school level. More specifically, 42% of 
teachers taught in 11 elementary schools, 19% in eight K-8 schools, 27% in 12 middle schools, 
and 13% in four grade 6-12 school (see Table 3.1). When examining participation by cohort, 64 
(31%) of the teachers were from Cohort 1 and 142 (69%) were from Cohort 2. 


Among the 206 teachers who completed the survey, 103 reported teaching at the 
elementary level, and 103 reported teaching at the secondary level. The secondary teachers 
reported teaching one to fourteen classes (Mean = 4.30). In addition, the secondary teachers 
reported that they used LDC materials in zero to eight classes (Mean = 2.54). Among secondary 
teachers, LDC was most commonly implemented in ELA (42%), social studies (36%), and science 
(15%), with a handful of teachers implementing in other subject areas such as mathematics 
(4%), the arts (4%), and Spanish (2%) (see Figure 3.1). 
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Table 3.1 
Number of Schools and Teachers Completing the Survey in 2017-2018 


Cohort/start year Elementary K-8 Middle 6-12 Total 
Number of schools 11 8 12 4 35 
Number of teachers 
In Cohort 1 schools and started 6 6 9 7 28 
2016-2017 
In Cohort 1 schools and started 2 0 10 2 14 
2017-2018 
In Cohort 1 schools and started 2 3 9 8 22 
2018-2019 
In Cohort 2 schools and started 34 18 16 8 76 
2017-2018 
In Cohort 2 schools and started 42 12 11 1 66 
2018-2019 
Total 86 39 55 26 206 
Figure 3.1 
Secondary Teachers’ Content Area Implementations 
50% 
oa 42% 
40% 
36% 
35% 
30% 
25% 
20% 
15% 
15% 
10% 
5% * 2% 
i i ar 
ELA (n = 43) Social Studies/ Science (n = 15) Math (n = 4) Arts (n = 4) Spanish (n = 2) 
History (n = 37) 
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LDC Participation (T1-T4) 


Of the responding teachers, 40% reported that 2018-2019 was their first year of 
implementing LDC, with the remaining 60% having one or 2 years of previous experience. 
Among those who were returning to LDC, a great deal of variation was reported about their 
module implementation prior to the 2018-2019 school year. More specifically, teachers 
reported implementing at least two modules (Mean = 2.69, Range = 0-15) and about three 
mini-tasks outside of the context of a module (Mean = 3.39, Range = 0-20) prior to 2018-2019. 


All 18 of the teachers who did not participate in a PLC did report using LDC tools in their 
instructional planning or classroom instruction during the 2018-2019 school year. Of the 17 
teachers who provided further information, the most common activities involved using 
CoreTools to access existing modules or mini-tasks (n = 13) and designing modules or mini tasks 
(n = 10). Others stated that they were given modules or mini-tasks by other teachers in their 
school (n = 7) or that they used the LDC online courses (n = 6). Finally, one teacher selected 
other and one additional teacher skipped the question. 


Professional Learning Community and Teacher Collaboration (T5-T9) 


The vast majority of teachers (90%) participated in a PLC that at least partly focused on 
implementing LDC. Most of these teachers reported that their PLC met once a week or more 
(55%) or every other week (21%). Only 24% of these teachers reported that their PLC met once 
a month or less. The most common reasons stated for not meeting every week included other 
priorities that competed with LDC (58.3%), PLC time not being protected (21%), and a failure by 
the school administrator to prioritize LDC (14%), or other reasons not already listed (25%). 
Other reasons listed primarily focused on scheduling or indicated that the timing of meetings 
was agreed upon by members of the PLC. The frequency of conversations about LDC that 
reportedly took place outside of the PLC meetings typically ranged from every other week to 
twice a week or more (70%). Two-thirds of respondents (66%) stated that PLC meetings 
typically lasted 45-59 minutes with another 16% stating that they lasted 60 minutes or more. 


LDC Training and Support (T10-T13) 


More than 80% of the teachers who responded stated that their PLC was moderately to 
very effective in all aspects listed on the survey. This included creating a comfortable working 
environment (92%), fostering an environment for sharing instructional plans (91%), helping 
teachers improve their LDC instructional plans (86%), and allowing space for the sharing of 
student work (90%). 


Teachers were asked their opinions of the online course materials in the LEARN tab in LDC 
CoreTools (see Figure 3.2). Overall, more than 60% of the teachers rated the materials as good 
to excellent on each aspect. Teachers were particularly enthusiastic about the clarity and 


? Three additional teachers checked "other" but indicated that it was not applicable (n = 1) or that they did actually 
meet each week (n = 2). 


39 


relevance of the information provided, as well as the opportunity to extend their learning, with 
about four out of five teachers rating these aspects as good or excellent. On the lower end, 
about one-third of teachers gave a rating of poor or fair concerning the usefulness of the 
videos. 


Figure 3.2 


Teacher Perceptions of Online Course Materials That LDC Coaches Used or Directed Their Teachers to Use 
(n = 187). 


ONLINE COURSE MATERIALS (TEACHER SURVEY) 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
Clarity of information presented. 
Relevance of information presented. 
Ease of use. 
Usefulness of resource documents. 


Usefulness of videos. 


Degree to which course materials helped teachers to 
create and/or adapt LDC modules. 


Opportunity to extend learning when needed or desired. 


mPoortofair Good to excellent 


Teachers were also asked to report on the supports that they received from their LDC 
coaches. When asked in general, almost all of the teachers reported that they were able to get 
feedback and support from their LDC coaches (93%) either through CoreTools or in the PLC, 
including the receipt of written feedback in a timely manner (96%) through CoreTools. Despite 
this, when asked to rate the usefulness of the coaching, seven teachers did indicate that they 
did not receive any written feedback from their coach within LDC CoreTools, 36 reported that 
they did not have any Zoom coaching sessions, and 36 indicated that did not communicate with 
their coach via email or phone. Of those who did receive the supports, most (79% to 82%) 
reported that they found it moderately or very helpful (see Figure 3.3). We also had 62 teachers 
indicate that they received other supports, with most of those who provided an additional 
comment indicating that they had in person meetings with their LDC coach.? 


3 Seven additional teachers provided a response for the other option, but their written comments focused on one- 
on-one conference calls via Zoom or phone (option 2), email (option 3), or stated not applicable. 
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Figure 3.3 
Teacher Perceptions of the Different Types of LDC Coaching Supports 


HELPFULNESS OF LDC COACH SUPPORTS (TEACHER SURVEY) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Written feedback in LDC CoreTools in the comments areas 
and/or via the teacher work rubric (n = 180) 


One-on-one Zoom video conference and/or call (n = 150) 


Email or phone communication (n = 151) 


Other (n = 62) 


Not ora little helpful m= Moderately or very helpful 


Figure 3.4. 
Extent to Which Teachers Completed Various Module Creation Tasks (n = 185) 


MODULE CREATION TASKS (TEACHER SURVEY) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Select focus standards for a writing assignment. 


Create a standards-driven writing assignment. 


Select high quality, complex texts and other materials to 
engage students in deeper learning. 


Identify the skills students need to develop to complete a 
writing assignment. 


Create daily lessons to teach the skills a student needs to 
complete a writing assignment. 


Differentiate instruction by incorporating multiple ways of 
thinking, various levels of complexity, and multiple... 


Plan for a variety of methods to assess student progress. 


Assess the quality of writing assignments and/or 
instructional plans using Peer Review/Curriculum... 


Make a writing assignment relevant and engaging for 
students. 


m Not at all to a little bit m= A moderate to a great extent 
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Module Creation (T14-T17) 


Teachers generally adapted (Mean 2.18, Range 0-16) more modules than they created 
(Mean = 1.43, Range = 0-8). New modules were most likely to be created by two or more 
teachers (58%), although some did report doing the work with their PLC as a whole (21%) or on 
their own (36%). In addition, six teachers noted that they only adapted modules at their PLC, 
three noted that it varies between individual and group work, and two noted that module 
development is only done by certain people. Most teachers (81% to 92%) also reported that 
they were able to accomplish most of the module creation tasks to a moderate or great extent. 
As can be seen in Figure 3.4, teachers generally felt most successful (a moderate extent toa 
great extent) in regards to selecting standards and creating standards-driven writing 
assignments, and in identifying the skills students need to develop to complete the writing 
assignment. The tasks that teachers reported doing the least (i.e., not at all or a little bit) 
included differentiating instruction and planning a variety of methods to assess student 
progress. 


Classroom Implementation (T18-T26) 


Teachers implemented more modules than stand-alone mini-tasks. More specifically, 
teachers reported implementing an average of about three modules (Mean = 2.75, 
Range = 0-15) during the school year, and four additional mini-tasks outside of the context of a 
module (M = 4.20, Range = 0-48). For 60% of the teachers this included implementing a “Find 
and Teach” module from CoreTools. Of those who completed this activity, about three-fourths 
reported adjusting the module (74%) and the remaining quarter (26%) reported implementing 
one of these modules without any changes. 


Teachers were also asked to talk about later modules they created or adapted after 
completing the Find and Teach. In this case, 42% of teachers reported starting from a template 
in CoreTools, with the remaining 58% adapting an existing module that they found in the LDC 
Library. Furthermore, 87% of teachers reported that they had already implemented the module 
by the time they completed their survey. The remaining teachers were planning to implement 
later during the school year (6%), during the next school year (3%), or were not planning to 
implement at all (4%). 


Finally, teachers were asked to report on what they were able to accomplish when 
teaching their LDC modules (see Figure 3.5). Overall, the vast majority of teachers (84% to 91%) 
reported accomplishing each of the activities listed to a moderate to great extent. The 
strongest opinions focused on accessing complex text (91%) as well as being able to engage 
students in understanding the assignment and its rubric, providing feedback using rubrics, and 
locating evidence of standards in student work (89%, respectively). The area where teachers 
seemed to have the most room to grow involved systematically collecting information about 
students’ progress with three reporting that they did not do this at all and 26 reporting that 
they only accomplished this a little bit. 
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Figure 3.5 
Activities Completed When Teaching LDC Modules (n = 183). 


CLASSROOM IMPLEMENTATION (TEACHER SURVEY) 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Engage students in understanding the assignment and its 
rubric. 
Engage students in accessing complex text for the purpose 
of the assignment. 
Systematically collect information about students’ 
progress. 


Provide feedback to students using assignment rubrics. 


Locate evidence of standards in final student work on the 
writing assignment. 
Use evidence of student progress on standards to modify 
subsequent instruction. 


m@ Not at all toa little bit = A moderate to a great extent 


Module Peer Review (T27-T29) 


Overall, the formal review process for modules seemed to be underutilized. First, only 
about one-quarter of the teachers (29%) who responded to the survey reported attending a 
peer review or curriculum alignment workshop during the 2018-2019 school year. Second, on 
average the teachers’ submitted less than one module each for the LDC National Peer Review 
process. Despite this, the 89 teachers who rated their experience tended to find the process 
moderately to very helpful (63%), and only 4% found the process to be not helpful. 


Impact on Teacher Practice and Learning (T30-T31) 


Almost all respondents reported that their teaching skills improved during the 2018-2019 
school year, and in most cases, this reflected moderate to a great deal of change (see Figure 
3.6). The skills with the most ratings of a great deal included creating standards-driven writing 
assignments (46%), selecting focus standards for a writing assignment (45%), and identifying 
the skills students need to develop to complete a writing assignment (44%). 


Teachers were also asked their perceptions of the impact of LDC using a 4-level 
agreement scale ranging from strongly disagree to strongly agree. As can be seen in Figure 3.7, 
the majority of teachers generally agreed or strongly agreed with the different impacts. More 
specifically, the vast majority of teachers agreed that LDC raised their expectations for students’ 
writing (88%), and helped them learn how to incorporate writing assignments (86%). 
Furthermore, while 83% of teachers reported that they were now more likely to collaborate 
when designing instruction, only 66% agreed that they shared their LDC work with colleagues 
who were not in their LDC PLC. 
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Figure 3.6 


Teacher Perceptions of the Impact of LDC on Their Teaching Practice and Learning (n = 183). 


IMPACT ON TEACHER SKILLS (TEACHER SURVEY) 


Selecting focus standards for a writing assignment. 
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a writing assignment. 
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complete a writing assignment. 
Systematically collecting information on students’ 
progress. 
Identifying patterns of student understandings or 
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Using evidence of student progress on standards to 
modify subsequent instruction. 


@ Not at all toa little 


0% 10% 20% 30% 40% 


m Moderately to a great deal 


50% 


60% 


70% 


80% 


90% 100% 


44 


Figure 3.7 
Teacher Perceptions of the Impact of LDC on Their Teaching Practice and Learning (n = 183) 


IMPACT ON TEACHER PRACTICE/LEARNING (TEACHER SURVEY) 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Participating in LDC raised my expectations for students’ 
writing. 
Using LDC modules became an important part of my 
instructional practice. 
Implementing LDC helped me incorporate my state's CCR 
standards into my instruction. 
LDC helped me incorporate writing assignments into my 
existing curriculum. 
lam more likely to collaborate with other teachers on 
designing instruction after participating in our LDC PLC. 


LDC helped me improve on my teacher evaluation ratings. 


Participating in LDC helped me develop working 
relationships with teachers in different grades and/or... 
| shared my LDC work with colleagues outside of the LDC 
PLC. 


m Strongly disagree or disagree m Agree or strongly agree 


Impact on Student Learning (T32) 


Teachers were asked about the impact of LDC on their students using a 4-level scale 
ranging from not at all to a great deal (see Figure 3.8). Overall, only a few teachers reported 
that their students received no positive effect in regards to any of the potential impacts listed 
(2% to 6%). The positive effects that teachers endorsed most strongly (i.e., rated as moderately 
or a great deal) focused on writing, the obtaining of content knowledge, and overall literacy 
performance. In contrast, teachers were more likely to say that their students only benefitted a 
little (19%, respectively) in regards to skills attainment (i.e., reading, college and career ready) 
and their ability to analyze and understand the components of a writing assignment. 
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Figure 3.8 
Teacher Perceptions of the Impact of LDC on Students (n = 183) 


IMPACT ON STUDENTS (TEACHER SURVEY) 
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Reading skills 

Content knowledge 

Ability to complete writing assignments 
Quality of students’ writing 


College and career ready skills 


Capacity to analyze and understand the components of a 
writing assignment 


Speaking and listening skills 


Overall literacy performance 


Performance on assessments throughout the school year 


wNotatalltoa little |§ = Moderately toa great deal 


Teacher Leader Support (T33) 


Teachers were asked to report on the support received from the teacher leaders in their 
respective PLCs. In this case, teacher opinions were overwhelmingly positive, with more than 
90% agreeing with each of the five types of support listed, and more than half strongly 
agreeing. Furthermore, 55% of the teachers strongly agreed that their they were comfortable 
approaching their teacher leader when they had questions about LDC, and 51% respectively 
strongly agreed that their teacher leader supported the PLC and/or was effective in inviting 
teachers to join LDC. Finally, half of the teachers strongly agreed that their teacher leader 
helped to align LDC with their instructional goals and/or provided useful feedback on modules. 


School Administrator Support (T34-T36) 


According to teachers, administrators varied in regards to how much they were engaged 
with the PLC. More specifically, 26% reported that their administrator attended the PLC 
meetings about three quarters or more of the time, 16% reported that their administrator 
attended about half of the meetings, and 58% reported that their administrator attended about 
one-quarter or less of the time. In addition, 38% of the teachers reported that their 
administrator never observed them teach an LDC mini-task, 25% reported that they were 
observed once, and 38% reported being observed two or more times. 
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Figure 3.9 
Teacher Perceptions of School Administrator Support (n = 183) 


SCHOOL ADMINISTRATOR SUPPORT (TEACHER SURVEY) 
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had a firm understanding of LDC. 
allocated resources...to ensure the LDC team could meet. 


encouraged teachers to participate in LDC. 


expressed concerns that implementing LDC is taking time 
away from other instructional priorities. 
communicated how using LDC’s tools supported specific 
school initiatives and/or goals. 
provided me with feedback about my LDC planning and/or 
instruction. 


made formative assessment a priority at my school. 


used LDC to implement standards-driven assignments 
within existing curriculum. 


m Strongly disagree to disagree m Agree to strongly agree 


Finally, teachers were asked to rate the support LDC received from school administrators 
using a 4-level agreement scale ranging from strongly disagree to strongly agree (see 
Figure 3.9). Teachers generally agreed or strongly agreed that their school administrators 
supported LDC. For example, 94% indicated that their administrator encouraged teachers to 
participate in LDC, and 86% each indicated that their administrator allocated resources to 
ensure that the LDC team could meet and/or made formative assessment a priority at the 
school. About half of teachers also disagreed or strongly disagreed that their principal 
expressed concerns about LDC taking time away from other instructional priorities. 


Teacher Leadership Role (T37) 


Next teachers were asked to report on their own roles as LDC leaders at their schools. As 
can be seen in Figure 3.10, about two-thirds of the teachers agreed or strongly agreed with 
each statement. The most commonly reported role involved helping to set instructional goals 
for the LDC work at their school. Furthermore, while 68% of teachers agreed or strongly agreed 
that they were interested in learning more about LDC implementation at their school by 
facilitating with their coach and/or colleagues, 16% strongly disagreed with this statement. 
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Figure 3.10 
Role of Teachers in the Leadership of LDC at Their School (n = 158) 


TEACHER LEADERSHIP ROLE (TEACHER SURVEY) 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


| was involved in setting instructional goals for the LDC 
work at my school. 
| was involved in discussions about how to expand LDC 
implementation at my school in future years. 

| had the opportunity to work with our LDC teacher leader 
and our administrator to help shape LDC implementation. 
| am interested in learning more about how to lead LDC 
implementation at my school by facilitating with the... 


m Strongly disagree to disagree m Agree to strongly agree 


Figure 3.11 
Teacher perceptions of facilitators and barriers to LDC implementation (n = 182) 


FACILITATORS AND BARRIERS (TEACHER SURVEY) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


My LDC PLC was given sufficient time to meet. 


| felt adequately prepared to effectively implement LDC 
modules in my classroom. 
It was challenging to find content-rich reading materials 
for the LDC modules | developed. 

My school had adequate technology to support teachers’ 
use of LDC. 

It was easy to find and adapt LDC mini-tasks for use in my 
classroom. 


m Strongly disagree to disagree m Agree to strongly agree 


Facilitators and Barriers (T38) 


Teachers were asked to rate their level of agreement with five potential facilitators or 
barriers to the implementation of LDC (see Figure 3.11). As would be hoped for, teachers 
generally showed positive agreement. More specifically, more than three quarters of the 
teachers indicated that they had sufficient time to meet in their PLC, that they felt adequately 
prepared to implement modules, had adequate technology to support the use of LDC, and had 
an easy time finding and adapting mini-tasks to implement. Finally, 21% of teachers strongly 
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agreed and 44% agreed that it was challenging to find content-rich reading materials to support 
their modules. 


3.2 Teacher Leader Survey Results 


In total, 30 teacher leaders spanning 25 schools completed the survey. 
LDC Participation (TL1-TL2) 


Only one of the 30 teacher leaders (3%) who responded to the survey reported that 
2018-2019 was their first year of implementing LDC. The remaining teacher leaders (97%) all 
had one or 2 years of previous experience with the program. Among those who were returning 
to LDC, a great deal of variation was reported about implementation during the 2016-2017 
and/or 2017-2018 school years. More specifically, teacher leaders reported implementing two 
to three modules (Mean = 2.68, Range = 0-6) and about four mini-tasks outside of the context 
of a module (Mean = 3.90, Range = 0-15). 


Professional Learning Community and Teacher Collaboration (TL3-TL6) 


Most of the teacher leaders reported that their PLC met every other week (37%) or once 
or more per week (40%). The most common reason stated for not meeting every week included 
PLC members having other priorities outside of LDC (61%) and PLC time not being protected 
(33%). A lot of variation was found concerning conversations outside of the PLC meetings, with 
60% reporting that these took place every other week or more and 40% reporting that they 
took place once a month or less. Finally, more than two-thirds of respondents (70%) stated that 
PLC meetings typically lasted 45-59 minutes, with another 10% stating that they lasted 60-74 
minutes. 


Teacher Training and Support (TL7-TL12) 


The vast majority of teacher leaders (87% to 93%) who responded stated that their PLC 
was moderately to very effective in all of the ways listed on the survey. This included creating a 
comfortable working environment, fostering an environment for sharing instructional plans, 
helping each other improve their LDC instructional plans, and sharing student work (see Figure 
3.12). 
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Figure 3.12 
Teacher Leader Perceptions About PLC Effectiveness (n = 30) 


EFFECTIVENESS OF PLC (TEACHER LEADER SURVEY) 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Creating an environment in which teachers are 
comfortable working together. 


Fostering an environment where teachers share their 
instructional plans with colleagues. 


Allowing space to share student work. 


Helping teachers learn to improve their LDC instructional 
plans. 


m Not effective to a little effective m Moderately to very effective 


Figure 3.13 
Teacher Leader Perceptions of the Online Course Materials in the LEARN Tab in LDC CoreTools (n = 30) 


ONLINE COURSE MATERIALS (TEACHER LEADER SURVEY) 
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Relevance of information presented. 


Ease of use. 
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Usefulness of videos. 


Degree to which course material helped teachers to create 
and/or adapt LDC modules. 


Opportunity to extend learning when needed 
or desired 


m Poor to fair Good to excellent 
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Teacher leaders were asked their opinions of the online course materials in the LEARN tab 
in LDC CoreTools (see Figure 3.13). Overall, more than two-thirds of the teacher leaders rated 
the materials as good to excellent. The most positive reactions were regarding the relevance of 
the information presented (93%), the usefulness of the resource documents (93%), and the 
clarity of the information presented (90%). The lowest ratings were for the usefulness of videos 
(67%) and the ease of use of the site (80%). Teacher leaders were also asked to report on the 
supports they received from their LDC coaches. When asked in general, almost all respondents 
said that they were able to get feedback and support from their LDC coach (97%) either through 
CoreTools or in the PLC, including the receipt of written feedback in a timely manner (97%) 
through CoreTools. Despite this, 7% reported that they did not receive written feedback in LDC 
CoreTools from their coach. Of those who did report receiving the different supports, views 
were positive with 82% or more of teacher leaders reporting them as moderately to very 
helpful (see Figure 3.14). These included written feedback, Zoom conferences, email or phone 
communication, or other sources of feedback. 


Figure 3.14 
Teacher Leader Perceptions of the Different Types of LDC Coaching Supports 


HELPFULNESS OF LDC COACH SUPPORTS (TEACHER LEADER 
SURVEY) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Written feedback in LDC CoreTools in the comments areas 
and/or via th teacher work rubric (n = 28) 


One-on-one Zoom video conference and/or call (n = 28) 


Email or phone communication (n = 30) 


Other (n = 10) 


m@ Not to alittle helpful m= Moderately to very helpful 


Finally, teacher leaders were asked seven positively oriented items and one negatively 
oriented item concerning the support they received from their LDC coach (see Figure 3.15). As 
would be hoped for, almost all teacher leaders (97% to 100%) agreed or strongly agreed to the 
positively oriented statements. In contrast, only 55% of teacher leaders disagreed or strongly 
disagreed that it was challenging to coordinate with their LDC coach on how to structure PLC 
time. 
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Figure 3.15 
Teacher Leader Perceptions of the Support They Received from the LDC Coaches 


SUPPORT RECEIVED FROM LDC COACH (TEACHER LEADER SURVEY) 
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| was able to reach my LDC coach if | had any questions 
about LDC. (n = 30) 


LDC provided adequate technical support for issues with 
the CoreTools online platform. (n = 28) 


LDC offered sufficient PD opportunities for me to lead the 
initiative in my school. (n = 29) 


LDC coaches were able to connect me with additional 
resources when needed. (n = 30) 


It was challenging to coordinate with our LDC coach on 
how to structure PLC time. (n = 29) 


When | reached out to our LDC coach, he or she 
responded quickly. (n = 30) 


Our LDC coach was easy to work with. (n = 30) 


Our LDC coach was knowledgeable and provided high 
quality guidance. (n = 30) 


m Strongly disagree to disagree m Agree to strongly agree 


Module Creation (TL13-TL16) 


Teacher leaders generally adapted twice as many modules (Mean = 4.10, Range = 0-15) as 
they created (Mean = 2.50, Range = 0-10). New modules were most likely to be created by 
teams of two or more teachers (67%), although many reported creating modules individually 
(47%) or in their PLC as a whole (27%). As can be seen in Figure 3.16, almost all teachers felt 
successful with regard to each of the module creation tasks. Furthermore, more than half of all 
respondents agreed to a great extent that they were able to select focus standards for a writing 
assignment, create a standards-driven writing assignment, and/or select high quality, complex 
texts, or other materials to engage students in deeper learning. Teacher leaders were a little bit 
less likely to be confident in their ability to differentiate instruction or to create daily lessons to 
teach the skills a student needs to complete a writing assignment. 
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Figure 3.16 
Teacher Leader Reports on Module Creation Activities (n = 30) 


MODULE CREATION ACTIVITIES (TEACHER LEADER SURVEY) 
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Select high quality, complex texts and other materials to 
engage students in deeper learning. 


Identify the skills students need to develop to complete a 
writing assignment. 


Create daily lessons to teach the skills a student needs to 
complete a writing assignment. 


Differentiate instruction by incorporating multiple ways of 
thinking, various levels of complexity, and multiple... 
Plan for a variety of methods to assess student progress. 
Assess the quality of writing assignments and/or 
instructional plans using Peer Review/Curriculum... 


Make a writing assignment relevant and engaging for 
students. 


m@ Not at all to a little bit m To a moderate or a great extent 


Impact on Student Learning (TL17) 


Teacher leaders were asked about the impact of LDC on student learning using a 4-level 
scale ranging from not at all to a great deal (see Figure 3.17). Almost all teacher leaders felt that 
students benefitted from LDC in each of the ways listed on the survey. The only item on which 
anyone reported “not at all” included speaking and listening skills (n = 1). In addition, one 
teacher leader each reported that their students only benefited a little regarding their reading 
skills, college and career ready skills, and speaking and listening skills. Finally, 53% of the 
teacher leaders reported that there was a great deal of impact on their students’ ability to 
complete writing assignments. 
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Figure 3.17 
Teacher Leader Perceptions of the Impacts of LDC on Student Learning (n = 30) 


IMPACT ON STUDENT LEARNING (TEACHER LEADER SURVEY) 
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School Administrator Support (TL18-TL19) 


According to teacher leaders, administrators at many of the schools were engaged with 
their PLC. More specifically, 21% reported that their administrator attended the PLC meetings 
about three quarters or more of the time, 14% reported that their administrator attended 
about half of the meetings, and 66% reported that their administrator attended on quarter or 
less of the time. 


Next, teacher leaders were asked to rate seven positively framed questions and one 
negatively framed question concerning the level of support that their school administrators 
provide to LDC (see Figure 3.18). Teacher leaders generally agreed or strongly agreed with each 
of the positively framed questions. For example, almost all respondents indicated that their 
administrator encouraged teachers to participate (90%), allocated resources to ensure the LDC 
team could meet (86%), communicated how LDC supports school initiatives/goals (86%), made 
formative assessment a priority (86%), and used LDC to implement standards-driven 
assignments (86%). Half of the teacher leaders (48%) also disagreed or strongly disagreed that 
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their administrator expressed concerns about LDC taking time away from other instructional 
priorities. 


Figure 3.18 
Teacher Leader Perceptions of School Administrator Support (n = 29) 


SCHOOL ADMINISTRATOR SUPPORT (TEACHER LEADER SURVEY) 
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had a firm understanding of LDC. 
allocated resources...to ensure the LDC team could meet. 


encouraged teachers to participate in LDC. 


expressed concerns that implementing LDC is taking time 
away from other instructional priorities. 


communicated how using LDC’s tools supported specific 
school initiatives and/or goals. 


provided me with feedback about my LDC planning and/or 
instruction. 


made formative assessment a priority at my school. 


Used LDC to implement standards-driven assignments 
within existing curriculum. 


m Strongly disagree to disagree m Agree to strongly agree 


Teacher Leader Leadership Role (TL20) 


Next, teacher leaders were asked to report on their own roles as LDC leaders at their 
schools. As can be seen in Figure 3.19, more than three-fourths of the teacher leaders (79% to 
97%) agreed or strongly agreed with each statement. The most commonly reported roles 
involved meeting regularly with their coach to manage the LDC work plan (97%), differentiating 
LDC implementation to meet the learning needs of teachers (93%), and building capacity as an 
instructional leader among colleagues (90%). The lowest levels of agreement involved leading 
the PLC without the assistance of the LDC coach (79%) and meeting regularly with the school 
administrator to make planning decisions around LDC (79%), which had three respondents 
strongly disagree. 
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Figure 3.19 
Leadership Roles of the Teacher Leaders (n = 29) 


LEADERSHIP ROLE (TEACHER LEADER SURVEY) 


| met regularly with my school administrator to make 
planning decisions around LDC. 


| was involved in discussions about differentiating LDC 
implementation to meet teacher learning needs. 


| was involved in discussions about how to expand LDC 
implementation at my school in future years. 


My role as an LDC lead teachers allowed me to effectively 
advocate for additional resources on my campus. 


| was involved in adjusting the problems of practice that 
my school targeted with the LDC work. 


| met regularly with my LDC coach to manage the LDC 
work plan. 


..allowed me to build my capacity as an instructional 
leader among my colleagues. 


| am confident that | can lead our LDC PLC in the future 
without the assistance of an LDC coach. 


m Strongly disagree to disagree 


Alignment (TL21) 


m Agree to strongly agree 


90% 100% 


Teacher leaders were asked to rate their agreement about the alignment of LDC with the 


goals and initiatives at their schools (see Figure 3.20). In this case, 62% to 93% of respondents 


agreed or strongly agreed with the six positively framed items. For example, more than 90% of 
respondents agreed that LDC is a strategy for implementing college and career ready standards, 


prepare students for current state assessments, and that LDC complements other school 
initiatives. The lowest levels of agreement were for the negatively framed items with 59% 
disagreeing that LDC interfered with other initiatives at the school and 38% disagreeing that it 
was difficult to focus on LDC because of competing school priorities. 
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Figure 3.20 
Teacher Leader Perceptions of the Alignment Between the School and LDC (n = 29) 


ALIGNMENT (TEACHER LEADER SURVEY) 
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school. 
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state assessments. 


It was difficult for teachers to focus on LDC because of 
other competing priorities at the school. 


Our instructional leaders are using LDC to implement 
standards-driven assignments within the existing 
curriculum. 


m Strongly disagree to disagree m Agree to strongly agree 


Scale-Up and Sustainability (TL22) 


Teacher leaders showed mixed levels of agreement to five questions that focused on 
issues of scale-up and sustainability (see Figure 3.21). Among the most positive results, 72% of 
respondents each indicated that they expected most teachers to continue implementing LDC 
the next year and/or believed that teachers and administrators were committed to sustaining 
LDC. In contrast, only 45% of teacher leaders agreed or strongly agreed that the LDC planning 
process and/or CoreTools was used by teachers at their schools who were not part of the PLC 
meetings. 
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Figure 3.21 
Teacher Leader Perceptions of the Scale-Up and Sustainability of LDC (n = 29) 


SCALE-UP AND SUSTAINABILITY (TEACHER LEADER SURVEY) 
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Figure 3.22 
Teacher Leader Perceptions of District Support of LDC (n = 37) 


DISTRICT SUPPORT (TEACHER LEADER SURVEY) 
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District professional development efforts were aligned 
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implementation of LDC. (n = 13) 


m Strongly disagree to disagree m Agree to strongly agree 


District Support (TL23) 


Overall, teacher leaders had positive attitudes regarding district support towards LDC (see 
Figure 3.22). More specifically, 83% of respondents agreed or strongly agreed that district 
leaders supported LDC implementation. In addition, 90% agreed that district leaders were 
interested in spreading its use to additional schools, 80% felt that the district leaders had a firm 
understanding of LDC, and 75% felt that district professional development efforts were aligned 
with LDC. Finally, only 62% reported that district leaders visited the school to discuss 
implementation. 


3.3 School Administrator Survey Results 


In total, 15 school administrators completed the survey, with one additional administrator 
answering only the first question. These administrators came from 16 different schools. Half 
were principals and half were assistant principals. 


Professional Learning Community (A2) 


Administrators varied in the proportion of the PLC meetings that they attended. More 
specifically, one third of administrators reported that they attended about three quarters of 
more of the meetings. In addition, 20% reported that they attended half of the meetings and 
another 20% reported attending about one quarter. Finally, 27% indicated that they attended 
less than one-quarter of the PLC meetings. It is not clear from the question whether any of the 
administrators failed to attend at all. 


Training and Support (A3) 


When asked about issues of training and support, all administrators who responded to the 
survey agreed or strongly agreed to each statement (see Figure 3.23). The strongest levels of 
agreement focused on whether the administrators were able to reach LDC staff when they had 
questions, adequate technology to access online resources, and whether their teacher leaders 
had sufficient PD opportunities (60%, respectively). Finally, about half of the administrators 
strongly agreed that LDC staff members were able to connect them with additional resources 
when needed and/or that there were sufficient PD opportunities for school administrators. 
Furthermore, while administrators attended an average of six in person and/or online sessions, 
some participated a great deal more or less (0-20). 
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Figure 3.23 
Administrator Perceptions of Training and Support 


TRAINING AND SUPPORT (ADMINISTRATOR SURVEY) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


| was able to reach LDC staff when | had questions about | ss 
LDC (n = 15). 


My school has adequate technology to access LDC online 
resources (n = 15). 


LDC offered sufficient professional development 
opportunities for LDC teacher leaders (n = 15). 


opportunities frschool administrators (v=), iii 

opportunities for school administrators (n = 15). 

detionel resources when needed (n= 14) i IE 
additional resources when needed (n = 14). 


mAgree mStrongly agree 
Note. One administrator responded disagree concerning whether their school had adequate technology. 


Classroom Observation (A5-6) 


All 15 of the administrators who completed the survey indicated that they observed each 
member of their PLC teaching an LDC module. Overall, the most common response among 
administrators was three or more observations per teacher (60%), with about one quarter 
observing twice (27%), and the remaining reporting observing each teacher once (13%). In 
addition, administrators were asked how effective they thought the modules they observed 
were at helping students develop literacy skills. All of these administrators perceived that the 
modules were moderately (60%) or very effective (40%). None of the administrators reported 
that the modules were not effective in helping develop students’ literacy skills. 


Impact on Teacher Practice (A7) 


Administrators were asked their perceptions about the level and types of impact that LDC 
had on teacher practice at their PLC (Figure 3.24). Overall, perceptions were positive with 80% 
to 100% indicating that LDC had a moderate to a great deal of impact on teachers. 
Administrators strongest perceptions focused on teachers being able to select focus standards 
for a writing assignment, creating standards-driven writing assignments, and systematically 
collecting information on students’ progress. The item with the least support involved teachers 
being able to identify patterns of students' understandings or misconceptions (80%). 
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Figure 3.24 
Administrator Perceptions of the Impact of LDC on Teacher Practice (n = 15) 


IMPACT ON TEACHER PRACTICE (ADMINISTRATOR SURVEY) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
Selecting a set of focus standards for a writing assignment. 


Creating standards-driven writing assignments. 


Identifying the skills students need to develop to complete 
a writing assignment . 

Creating daily lessons to teach each skill a student needs 
to complete a writing assignment. 

Systematically collecting information on students’ 

progress. 
Identifying patterns of student understandings or 
misconceptions. 

Using evidence of student progress on standards to 

modify subsequent instruction 


mNotatalltoalittle © Moderately toa great deal 


Figure 3.25 
Administrator Perceptions of the Impact of LDC on Student Learning (n = 15) 


IMPACT ON STUDENT LEARNING (ADMINISTRATOR SURVEY) 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
Reading skills 
Content knowledge 
Ability to complete writing assignments 
Quality of students’ writing 
College and career ready skills 
Capacity to analyze and understand the components of a... 
Speaking and listening skills 
Overall literacy performance 


Performance on assessments throughout the school year 


mNotatalltoalittle =m Moderately to a great deal 
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Impact on Student Learning (A8) 


Administrators also perceived a variety of positive impacts on the students whose 
teachers implemented LDC (Figure 3.25). For example, 87% each indicated that LDC had a 
moderate to a great deal of impact on their students’ reading skills, ability to keep writing 
assignments, speaking and listening skills, and overall literacy performance. The area where 
administrators were least likely to perceive a moderate or greater impact (73%) involved the 
capacity for students to analyze and understanding the components of a writing assignment. 


Administrator Leadership Role (A9) 


Next, administrators were asked to report on their own roles as LDC leaders at their 
schools. As can be seen in Figure 3.26, the vast majority (80% to 93%) of administrators who 


responded agreed or strongly agreed with each statement. The most commonly reported role 
(93%) involved discussions about differentiating LDC implementation to meet teacher learning 
needs. In addition, 87% agreed or strongly agreed that they met regularly with the LDC teacher 
leader, led discussions about how to expand implementation in future years, made changes to 
school schedules to accommodate PLC time, and allocated resources to ensure the team could 
meet. Finally, the least common role (80%) involved being able to shape implementation of LDC 


at his or her school. 


Figure 3.26 
Administrator Perceptions of Their Leadership Role with LDC (n = 15) 


ADMINISTRATOR LEADERSHIP ROLE (ADMINISTRATOR SURVEY) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


| was able to shape LDC implementation at my school. 


| met regularly with the LDC teacher leader in my school to 
stay abreast of implementation progress. 
| was involved in discussions about differentiating LDC 
implementation to meet teacher learning needs. 
| led discussions about how to expand my school’s LDC 
implementation in future years. 
| made changes to school schedules to accommodate LDC 
professional learning time. 
| allocated resources such as teacher time, payment, 

administrator time, support staff, sub coverage, etc., to... 


m Strongly disagree to disagree m Agree to strongly agree 
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Alignment (A10) 


Administrators were also asked to rate their agreement about the alignment of LDC with 
the goals and initiatives at their schools (see Figure 3.27). In this case, 80% or more of the 
administrators agreed or strongly agreed with the seven positively framed statements. For 
example, all administrators agreed or strongly agreed that LDC helped teachers create writing 
assignments to use within their current curricula. In addition, 93% of respondents agreed that 
LDC complemented other school initiatives, is a strategy for implementing college and career- 
ready standards, that they were using it to implement standards-driven assignments, and that it 
prepares students for current state assessments. finally, only one-third of the administrators 
agreed with the two negatively framed items, i.e., that time implementing LDC interfered with 
other important initiatives at the school and that it was difficult to focus on LDC because of 
other school priorities. 


Figure 3.27 
Administrator Perceptions of LDC Alignment with the School (n = 15) 


ALIGNMENT (ADMINISTRATOR SURVEY) 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Our school connected LDC implementation to our specific 
schoolwide goals. 


LDC helped teachers create writing assignments to use 
within their current curricula. 


LDC complemented other initiatives taking place in my 
school. 


| view LDC as a strategy for implementing my state’s 
College- and Career-Ready Standards. 


The time spent implementing LDC interfered with other 
important initiatives at my school. 


LDC helped prepare students in my school for current 
state assessments. 


It was difficult for teachers to focus on LDC because of 
other competing priorities at the school. 


| am using LDC to implement standards-driven 
assignments within the existing curriculum. 


LDC helped improve teacher evaluation ratings. 


m Strongly disagree to disagree m Agree to strongly agree 


Scale-Up and Sustainability (A11) 


Administrators varied regarding to whether they felt qualified to answer each of the five 
scale-up and sustainability questions (see Figure 3.28). For example, only eleven administrators 
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expressed an opinion about whether teachers would participate in LDC again next year. Despite 
this, of those who did respond, opinions were normally very positive. The vast majority agreed 
that they and their teachers were committed to sustaining LDC, that teachers who were already 
participating would continue, that collaboration was improved across grades and/or subjects as 
a result of LDC, and that the PLC would increase in size next year. Finally, only 42% reported 
that teachers who were not part of the LDC PLC meetings were using the LDC planning process 
or CoreTools web site. 


Figure 3.28 
Administrator Perceptions of the Scale-Up and Sustainability of LDC 


SCALE-UP AND SUSTAINABILITY (ADMINISTRATOR SURVEY) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


| expect that most teachers participating in LDC this year 
will continue to do so next year. (n = 11) 


Teachers at my school who were not part of the LDC PLC 
meetings used the LDC planning process and/or LDC 
CoreTools. (n = 12) 
As a result of LDC, new collaborations across grades 
and/or subjects were created or are being launched at my 
school. (n = 13) 


Teachers and administrators at my school are committed 
to sustaining the LDC initiative. (n = 12) 


| expect our LDC PLC to increase in size next year. (n = 11) 


m Strongly disagree to disagree m Agree to strongly agree 


District Support (A12) 


Similarly, administrators varied regarding whether they felt qualified to answer each of 
the five district support items listed in Figure 3.29. For example, only five expressed an opinion 
about whether district leaders were interested in spreading the use of LDC to additional 
schools. Despite this, of those who did respond, opinions were generally very positive. More 
specifically, more than three-quarters of the respondents agreed or strongly agreed that district 
leaders supported implementation, that district leaders had a firm understanding of LDC, and 
that district PD was aligned with the LDC initiative. Finally, only 45% or five administrators 
agreed that district leaders visited their school to discuss implementation of LDC. 
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Figure 3.29 
Administrator Perceptions of District Support of LDC 


DISTRICT SUPPORT (ADMINISTRATOR SURVEY) 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


District leaders supported the implementation of LDC. (n= 
9) 


District leaders had a firm understanding of LDC. (n = 8) 


District leaders are interested in spreading the use of LDC 
to additional schools. (n = 5) 
District professional development efforts were aligned 
with the LDC initiative. (n = 9) 
District leaders visited my school to discuss the 
implementation of LDC. (n= 11) 


m Strongly disagree to disagree m Agree to strongly agree 


3.4 Open-Ended Responses for All Participants 


Each survey respondent was asked three open-ended questions regarding the supports 
they received for the implementation of LDC at their school: (a) What supports did you find the 
most useful and why? (b) What supports were not helpful and why? (c) In what ways could LDC 
implementation be improved in your school in the future? The following section summarizes 
the findings for the 158 teachers, 28 teacher leaders, and 14 administrators who responded to 
the open-ended questions. 


Results were analyzed thematically for the supports of coaching, CoreTools, and 
professional development, all of which were mentioned in the item prompt. In addition, we 
coded responses for collaboration and other more general issues that did not clearly fit within 
the other major themes. For example, responses that mentioned meetings but did not explicitly 
state whether they were for coaching, collaboration, or professional development. Finally, we 
coded responses that did not answer the question, for example, responses that said not 
applicable or nothing, people who only made negative comments when asked to say what was 
most useful, and individuals who only said something positive in regards to the question about 
what was least helpful. 


General Views 


About one-quarter of the open-ended responses (136 comments) did not directly respond 
to the question being asked. For example, 107 of the responses across the three questions had 
a neutral statement such as “not applicable” or “nothing.” In addition, 22 of the responses to 
the question about what was least helpful and six of the responses to the question asking for 
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suggestions simply provided a positive comment such as noting that the supports were 
“helpful.” Only one response across the three questions made a negative statement. 


Table 3.2 


Major Themes from Open-Ended Survey Items about LDC Supports 


Teacher Teacher leader Administrator Total 
Supports # % # % # % # % 
MOST USEFUL 
Collaboration 11 7.0 0 0.0 2 14.3 13 6.5 
Coaching 92 58.2 23 82.1 11 78.6 126 63.0 
CoreTools 57 36.1 8 28.6 8 57.1 73 36.5 
PD 17 10.8 6 21.4 1 7.1 24 12.0 
Other 15 9.5 1 3.6 0 0.0 16 8.0 
LEAST USEFUL 
Collaboration 3 1.9 0 0.0 0 0.0 3 1.5 
Coaching 30 19.0 4 14.3 2 14.3 36 18.0 
CoreTools 35 22.2 5 17.9 2 14.3 42 21.0 
PD 9 5.7 6 21.4 1 7.1 16 8.0 
Other 18 11.4 5 17.9 0 0.0 23 11.5 
SUGGESTIONS 
Collaboration 18 11.4 5 17.9 0 0.0 23 11.5 
Coaching 16 10.1 3 10.7 3 21.4 22 11.0 
CoreTools 22 13.9 4 14.3 1 7.1 27 13.5 
PD 13 8.2 0 0.0 2 14.3 15 7.5 
Other 80 50.6 15 53.6 7 50.0 102 51.0 


Note. Each comment was coded for one or more themes; therefore, percentages exceed 100%. 


Collaboration 


While not mentioned in the section prompt, a small number of respondents brought up 
collaboration as a form of support that they considered helpful (13 comments) or that they had 
suggestions about for the future (23 comments). Specific themes included the value of getting 
to engage with their colleagues to work together on modules, but also to provide feedback and 
suggestions, and discuss issues. A number of teachers also specifically mentioned the 
usefulness of the PLC meetings for the purpose of collaboration. In regards to suggestions, 
respondents wanted more opportunities to co-plan, share what they learned, observe other 
teachers, and provide more of a balance in focusing on modules versus student work. A few 
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respondents also mentioned that they would like the opportunity to collaborate across subject 
areas or grade levels. Furthermore, some teachers wanted more in person PLC time. Finally, in 
response to the question about what was not helpful, two teachers indicated that the PLC 
meetings were not always productive, and another noted that they were not given enough time 
to co-plan and check-in with their grade level team. 


Coaching 


Based on the open-ended responses, coaching was the most valued type of support 
provided for the implementation of LDC (126 comments). The majority of these comments 
focused on one or more of the modes in which the coaching was provided. More specifically, 26 
respondents valued the in-person coaching, 63 valued the virtual coaching, and 19 valued 
coaching through both of these modes. In some cases, respondents provided extra detail, with 
32 noting that they valued using Zoom for the virtual coaching and eight mentioning the use of 
email. In addition, one respondent noted that they liked the group meetings, and another three 
pointed out that they preferred to receive their coaching one-on-one. A moderate number of 
respondents to this question also mentioned the person who provided the coaching for their 
school (55 comments). For example, respondents mentioned their LDC coach, teacher leader, 
district lead, or provided the name of an individual who provided coaching. Finally, 25 
respondents pointed out that one of the benefits of the coaching was the opportunity to 
receive written or verbal feedback. 


Participants also provided comments about what they felt did not work well or what they 
would like to have seen regarding the coaching. The most common theme regarding what did 
not work well involved the virtual coaching (34 comments). Individuals particularly felt that 
Zoom was problematic (25 comments) with some pointing out that they had technology issues 
and/or that they were uncomfortable with the pacing or timing of these coaching sessions. 
Three respondents also mentioned that they did not find the in person coaching helpful, two 
did not like the one-on-one coaching, and three did not like the group coaching sessions. 


The results for the question about suggestions continued many of these themes. That is, 
we had 13 individuals suggested having more or adding in person coaching, and one suggested 
having more Zoom meetings. Four respondents were more specific and mentioned their 
teacher leader or coach, and one mentioned that they needed a new coordinator to continue 
coaching during the next school year. Finally, two respondents suggested what type of content 
they would like covered during coaching sessions, and one respondent suggested a change in 
scheduling. 


CoreTools Online Platform 


Based on their comments, teachers, teacher leaders, and administrators all felt that there 
were positive (73 comments) and less positive aspects to the CoreTools platform (42 
comments). While some of the respondents who liked the CoreTools did not specify why it was 
a helpful support, those who did were most likely to highlight the library (18 comments) that 
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includes different mini-tasks and modules that can be integrated or adapted, the module 
creator (17 comments), and the LEARN tab (14 comments) that includes various professional 
development materials. Small numbers of respondents also noted that they liked the feedback 
features in the CoreTools (8 comments). 


For those respondents who felt that there were drawbacks to the CoreTools, the most 
cited issues involved the ease of use of the platform (16 comments), the LEARN tab 
(9 comments), and the library (8 comments). In addition, seven respondents noted that they 
did not find all or part of the module creator to be helpful. Suggestions that respondents made 
focused on similar themes including the library (9 comments), the ease of use of the platform (7 
comments), and the Module Creator (4 comments). In addition, one respondent suggested a 
change concerning the LEARN tab and another suggested a change concerning module 
implementation. Among the specific suggestions made were the desire to have more standard 
specific modules, scoring rubrics for mini-tasks, and making sure that the texts and topics in the 
library are appropriate for students. 


Professional Development 


Respondents also tended to point out more positives (24 comments) than negatives (16 
comments) regarding the professional development they were offered as part of their 
implementation of LDC. For the most part this difference was driven by opinions about the in 
person professional development, with 11 respondents finding it helpful and only one person 
finding it to be not helpful. In contrast, numbers were similar with other issues. For example, 
three respondents each liked or did not like the content of the PD, four respondents liked and 
three disliked the summer session(s), and seven liked and eight disliked the LEARN tab courses. 
Finally, three respondents noted that they liked the individual who provided the professional 
development and one respondent noted that they were unsatisfied with the amount of PD and 
would have liked more. 


Suggestions provided by teachers and administrators covered most of the same themes 
concerning professional development. For example, four noted that they would like more in 
person professional development opportunities, and one noted that they would like these in 
person opportunities to be more hands on. Five respondents noted that they wanted more 
professional development opportunities, and two noted that they wanted more experiences 
with the LEARN tab courses. Finally, two respondents talked about when they would like 
training (i.e., summer, out of school time), and two others made suggestions about the content 
of these trainings (i.e., navigating CoreTools, leadership). 


Other General Views 


As previously noted, some responses did not clearly align with the four main themes. 
While these were mostly found in response to the question asking for suggestions (102 
comments), some were found for the other two questions (39 comments). The most common 
themes found across the three questions focused on issues of time (45 comments) and/or 
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meetings (24 comments). More specifically, individuals noted that they liked the in-person 
meetings, and some noted that the main drawback was that they did not spend more time 
meeting, using the LEARN tab, or implementing LDC in general. Similar issues were brought up 
in the suggestions concerning meetings (17 comments) and time (38 comments), with most 
focused on wanting more meeting time, more planning time, or time for LDC in general. 
Moderate numbers of respondents also voiced their interest in having better alignment 
between LDC and the district or school curricula (17 comments), their desire to expand 
participation to other grades or subjects (15 comments), and some general issues of support 
(12 comments). Ten of the participants also voiced that they would like more or better support 
from their administrators for LDC. Finally, some respondents would like to have seen more 
attention paid to English language learners (ELLs), special needs students, or 
primary/elementary school students. 


3.5 Exploratory Analysis 


To examine change in perceived impacts among returning teachers, survey ratings were 
compared for teachers who submitted a survey during both spring 2018 and spring 2019. The 
following presents the results for the 83 teachers who participated at both time points. 


When examining the longitudinal results regarding the impact of LDC on their teaching 
skills, the percentage of teachers who provided a positive rating of moderately to a great deal 
increased very slightly across the two school years (see Figure 3.30). More specifically, 
percentages increased by 1% to 5% for each item. Furthermore, by spring 2019, the percentage 
of teachers providing positive ratings met or exceeded 92% on all items. In addition, as can be 
seen in Table 3.3, when looking at the direction of change in ratings across the four-point scale 
(not at all to a great deal), most teachers ratings showed no change (55% to 71%) or increased 
by one or more levels (14.5% to 26.5%). 


Table 3.3 
Change in Teacher Perceptions of the Impact of LDC on Teacher Skills (n = 83) 


Negative No Change Positive 


Dimension 
# % # % # % 


Selecting focus standards for a writing assignment. (1) 10 12.0 59 71.1 #14 16.9 
Creating standards-driven writing assignments. (2) 12 145 59 71.1 12 14.5 


Identifying the skills students need to complete a 13 15.7 48 57.8 22 26.5 
writing assignment. (4) 


Creating daily lessons to teach the skills students need 16 19.3 46 55.4 21 25.3 
to complete a writing assignment. (5) 


Note. Four level agreement scale ranging from not at all to a great deal. 
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Figure 3.30 
Change in Teacher Perceptions of the Positive Impact of LDC on Teacher Skills (n = 83) 


CHANGE IN POSITIVE IMPACT ON TEACHER SKILLS (TEACHER SURVEY) 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Selecting focus standards for a writing assignment. 


Creating standards-driven writing assignments. 


Identifying the skills students need to develop to complete 
a writing assignment. 


Creating daily lessons to teach the skills students need to 
complete a writing assignment. 


m2018 2019 


Note. Only four items were asked concerning teacher skills on the survey during both 2018 and 2019. 


Figure 3.31 


Change in Teacher Perceptions of the Positive Impact of LDC on Teaching Practice and Learning (n = 83) 


CHANGE IN POSITIVE IMPACT ON TEACHER PRACTICE/ LEARNING (TEACHER SURVEY) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Participating in LDC raised my expectations for students’ 
writing. 


Using LDC modules became an important part of my 
instructional practice. 


Implementing LDC helped me incorporate my state's CCR 
standards into my instruction. 


LDC helped me incorporate writing assignments into my 
existing curriculum. 


lam more likely to collaborate with other teachers on 
designing instruction after participating in our LDC PLC. 
LDC helped me improve on my teacher evaluation ratings. 
Participating in LDC helped me develop working 
relationships with teachers in different grades and/or... 


| shared my LDC work with colleagues outside of the LDC 
PLC. 


m2018 2019 
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Positive growth was generally found when examining change in teachers’ perceptions 
about the impact of LDC on various facets of their teaching practice and learning (see Figure 
3.31). For example, percentages increased by 3% to 11% on six of the items, with the most 
growth being found regarding incorporating state standards into instruction, LDC modules 
being an important part of their instructional practice, and improving their teacher evaluation 
ratings. The only items in which percentages dropped slightly (1 to 3%) focused on 
collaboration. Furthermore, by Spring 2019, the percentage of teachers providing positive 
ratings exceeded 75% on all but one item. Finally, as can be seen in Table 3.4, when looking at 
the direction of change in ratings across the four-point scale (not at all to a great deal), most 
teachers ratings showed no change (54% to 64%) or increased by one or more levels (20% to 
32%). 


Table 3.4 


Change in Teacher Perceptions of the Impact of LDC on Teacher Practice and Learning, Counts and Percentages 
(n = 83) 


Negative No Change Positive 

Dimension # % # % # % 
Participating in LDC raised my expectations for 11 13.3 45 54.2 27 32.5 
students’ writing. 
Using LDC modules became an important part of my 7 8.4 50 60.2 26 31.3 
instructional practice. 
Implementing LDC helped me incorporate my state's 9 10.8 49 59.0 25 30.1 
CCR standards into my instruction. 
LDC helped me incorporate writing assignments into 12 14.5 48 57.8 23 27.7 


my existing curriculum. 


lam more likely to collaborate with other teachers on 13 15.7 49 59.0 21 25.3 
designing instruction after participating in our LDC 


PLC. 

LDC helped me improve on my teacher evaluation 11 13.3 47 56.6 25 30.1 
ratings. 

Participating in LDC helped me develop working 13 15.7 53 63.9 17 20.5 
relationships with teachers in different grades and/or 

subjects. 

| shared my LDC work with colleagues outside of the 14 16.9 47 56.6 22 26.5 
LDC PLC. 


Note. Four level agreement scale ranging from strongly disagree to strongly agree. 
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Figure 3.32 
Change in Teacher Perceptions of the Positive Impact of LDC on Students (n = 83) 


Change in Positive Impact on Students (Teacher Survey) 


0% 10% 20% 30% 40% 50% 60% 70% 80% 
Reading skills 
Content knowledge 
Ability to complete writing assignments 
Quality of students’ writing 
College and career ready skills 
Capacity to analyze and understand the components of a... 
Speaking and listening skills 
Overall literacy performance 


Performance on assessments throughout the school year 


2018 2019 


Table 3.5 


Change in Teacher Perceptions of the Impact of LDC on Students, Counts and Percentages (n = 83) 


90% 100% 


Negative No Change Positive 
Dimension # % # % # % 

Reading skills 15 18.1 46 55.4 22 26.5 
Content knowledge 11 13.3 47 56.6 25 30.1 
Ability to complete writing assignments 10 12.0 48 57.8 25 30.1 
Quality of students’ writing 9 10.8 50 60.2 24 28.9 
College and career ready skills 12 14.5 47 56.6 24 28.9 
Capacity to analyze and understand the components 13 15.7 52 62.7 18 21.7 
of a writing assignment 

Speaking and listening skills 11 13.3 42 50.6 30 36.1 
Overall literacy performance 14 16.9 50 60.2 19 22.9 
Performance on assessments throughout...school year 11 13.3 47 56.6 25 30.1 


Note. Four level agreement scale ranging from not at all to a great deal. 
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Finally, we examined changes in teacher ratings concerning the impact of LDC on nine 
different student outcomes. As can be seen in Figure 3.32, positive ratings (moderately to a 
great deal) by teachers generally stayed the same or increased slightly between spring of 2018 
and spring of 2019. More specifically, the largest growth in positive ratings was found for 
students’ college and career ready skills (5%) and speaking and listening skills (5%). In contrast, 
ratings went down by 1% concerning students’ capacity to analyze and understand the 
components of a writing assignment, and showed no change concerning reading skills, or 
regarding students' performance on assessments throughout the school year. Finally, results 
concerning general changes in ratings mirrored those found for the other two longitudinal 
analyses (see Table 3.5). More specifically, the largest proportion of teachers showed no 
change in their ratings for the items between the two survey administrations (51% to 63%), or 
showed an increase of one or more levels on the agreement scale per item (22% to 36%). 


3.6 Summary of Results 


This survey captured the responses of 75% of all participants in the district’s third year of 
implementation. Based on the survey analyses, a majority of teachers and teacher leaders 
implemented LDC as expected. Most reported implementing multiple modules across the 
school year. The majority of participants also reported that they developed or adapted their 
module(s) collaboratively and that they primarily adapted existing modules. In their open- 
ended responses, 43% of participants also noted that they found the CoreTools platform and/or 
professional development to be helpful. Despite this, some did comment that the CoreTools 
site in general and the LEARN Tab in specific could be difficult to navigate, and one-third rated 
the videos in the LEARN tab as less useful. In addition, while the teachers who made use of the 
National Peer Review process found it helpful, most teachers reported that they did not make 
use of this LDC feature. 


Participants tended to feel supported in their implementation of LDC. At the local level, 
teachers had very positive opinions of their teacher leaders, frequently reported that their 
administrator(s) provided encouragement and resources, and some felt that the opportunity to 
collaborate with their peers was helpful. Most of the teachers and teacher leaders also tended 
to report positive opinions of the LDC coaches, and more than 60% noted that the coaching was 
one of their most valued supports. In addition, while teacher leaders and administrators felt 
that the district supported implementation of LDC, moderate percentages reported that district 
leaders did not visit their school to discuss implementation of LDC. Some participants also 
expressed concerns about whether participants in the PLC were committed to sustaining LDC, 
with the most commonly noted issues involving the lack of use by teachers outside of the PLC 
and worries about whether new teachers would be recruited into the PLC during the next 
school year. 


Participants perceived multiple positive impacts of the LDC initiative that aligned with the 
initiative. For example, most teachers, teacher leaders, and administrators felt that the 
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program helped students to develop their writing and literacy skills. Teachers and 
administrators felt that teachers improved their ability to select focus writing standards and 
create standards-driven writing assignments. In addition, most teachers reported improvement 
in their expectations for their students’ writing, their ability to incorporate writing assignments 
and state standards into the curriculum, and their likeliness to collaborate on the design of 
instruction. The areas with the least perceived impact for students involved reading skills and 
college and career ready skills. Respondents also had slightly lower perceptions of the impact of 
LDC on teachers' skill in systematically gathering evidence of student progress and their ability 
to identify patterns in student understandings and misconceptions. Finally, less than 75% of 
teachers reported that their participation impacted their teacher evaluation ratings or their 
practice of sharing LDC work with their non-participating colleagues. 


Finally, an exploratory analysis was conducted to examine change in teachers' perceptions 
of the impacts of LDC for the 83 teachers who submitted a survey in both spring 2018 and 
spring 2019. As with the primary analyses for the three scales we examined (teacher skills, 
teacher practice and learning, and student skills) we primarily found small, but positive results. 
More specifically, the percentage of teachers who provided positive ratings increased for 15 of 
the items, stayed the same for two, and decreased slightly for four items. The greatest 
increases were found for teachers’ perceptions of their ability to incorporate standards into 
instruction (+11%) and improve their teacher evaluation ratings (+8%), and the largest amounts 
of negative change (-3%) were found for collaboration with other teachers on designing 
instruction. Finally, when examining general change in ratings across all levels of the agreement 
scales, we found that most teachers provided the same or a higher rating. In other words, 
teachers tended to maintain or improve in their positive perceptions about the impact of LDC 
on their own practice, as well as on their students’ skills. 
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4.0 Analyses of LDC CoreTools Data 


In the following section, we report on how participants interacted with LDC’s CoreTools 
online system. We begin by presenting participation rates for key CoreTools activities, including 
creating a user account, and viewing, editing, and commenting on modules. We then dig 
deeper into the three CoreTools activities, by providing descriptive statistics for all i3 CoreTools 
users followed by results broken down by key subgroups (including LDC role, school level, 
cohort, and content area taught). After that, we compare the level of engagement for i3 
CoreTools users who completed and taught a full-length module to those users who did not 
complete and teach a module during the school year. Finally, we summarize results. 


4.1 CoreTools Activity Participation Rates 


The four key CoreTools activities we examined include creating a user account as well as 
viewing, editing, and commenting on modules. Nearly all teachers and teacher leaders created 
CoreTools user accounts, although administrators not acting as teacher leaders did so ata 
somewhat lower rate (81%). Among the 289 CoreTools users, 250 were classroom teachers 
(with 32 of those individuals acting as teacher leaders), three were coaches or coordinators 
playing the teacher leader role, two were administrators playing the administrator role, and 42 
of them were administrators overseeing the LDC work but not acting as teacher leaders. 


Table 4.1 
CoreTools Feature Participation Rates: 2018-2019 (Counts and Percentages) 


Gacicosneiene N User Viewed a Edited a Commented 
P YP account module module ona module 
Teacher 222 218 (98%) 207 (93%) 169 (76%) 97 (44%) 
Teaenenlege cr icassioam 32 32(100%)  31(97%)  29(91%) 25 (78%) 
teacher) 
Total teachers 254 250 (98%) 238(94%) 198 (78%) 122 (48%) 
Teacher leader 
0, 0, 0, 0, 
ee en 3 3 (100%) 3 (100%) 3 (100%) 3 (100%) 
Teacher leader (administrator) 2 2 (100%) 2 (100%) 1 (50%) 0 (0%) 


Administrators (principals and 
assistant principals, not acting 42 34 (81%) 12 (29%) 2 (5%) 2 (5%) 
as teacher Leaders) 


Total participants 301 289 (96%)  255(85%) 204 (68%) 127 (42%) 


As seen in Table 4.1, most participants used CoreTools to at least some degree. Ninety-six 
percent of all participants created a user account, 85% of participants viewed modules, and 
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68% edited modules. Commenting on modules was a less common activity with only 42% of all 
participants doing so. 


As previously noted, nearly all participants created a CoreTools user account. Significantly, 
even administrators, who generally were not a part of the regular PLC meetings, created user 
accounts. This suggests that the vast majority of administrators overseeing LDC PLCs had at 
least some familiarity with the online platform. Likewise, nearly all PLC teachers and teacher 
leaders viewed modules in CoreTools. A little less than one third of principals and assistant 
principals viewed modules in the platform. 


We consider editing modules to be the key indicator of deep engagement with the 
CoreTools module building platform. Seventy-eight percent of participating teachers edited at 
least one module. In contrast, only three administrators engaged at the level of editing 
modules. 


Overall, adding comments to modules was a much less common activity. About half of 
classroom teachers and only five coaches, coordinators, or administrators commented. 
Commenting was, not surprisingly, most common among teacher leaders, whose role was most 
conducive to giving feedback to PLC members. 


4.2 Engagement with Key CoreTools Activities 


In this section, we describe participants’ level of engagement with three key CoreTools 
activities: viewing, editing, and commenting on modules. Descriptive statistics are reported for 
all participants, as well as a number of subgroups. These subgroups capture the role the 
individual played in LDC (teacher, teacher leader, and administrator), the level of the school at 
which the participant worked (elementary, K-8, middle, or 6-12), the cohort group describing 
when the school and individual participant began taking part in LDC, and in the case of 
teachers, the content area taught (elementary/multiple subjects, secondary ELA, secondary 
history/social studies, and secondary science). As noted earlier, some participants played 
multiple roles in the intervention, so the teacher, teacher leader, and administrator groups 
overlap to some degree as they do in our survey analysis. Descriptive statistics are only 
reported for groups with five or more members. The samples for the viewing, editing, and 
commenting analyses are the 289 LDC participants with CoreTools user accounts in 2018-2019 
(see Table 4.1). 


Module Viewing 


In Table 4.2 we present descriptive statistics on how many times i3 participants viewed 
modules, both overall and by subgroup. We present the minimum and maximum number of 
views, means and standard deviations, and the sum of total views across participants. Overall, 
the average participant viewed modules about 41 times, although the range varied greatly from 
zero to 965 views. There were 34 participants with user accounts who did not view any 
modules (about 12% of the overall sample). 
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Table 4.2 


Descriptive Statistics for the Number of Times a Participant Viewed a Module in CoreTools, by Participant 


Subgroup 
Subgroup N Min Max Mean SD Sum 
All participants 289 0 965 41.3 86.2 11,941 
Participant role 
All teachers 250 0 732 43.1 70.3 10,762 
All teacher leaders 37 0 965 97.6 191.1 3,611 
All administrators 36 0 33 2.7 6.2 96 
Cohort 
Cohort 1 / participant started 2016-2017 40 0 732 68.0 146.2 2,720 
Cohort 1 / participant started 2017-2018 18 2 206 35.9 45.3 647 
Cohort 1 / participant started 2018-2019 27 3 127 37.9 32.9 1,024 
Cohort 2 / participant started 2017-2018 120 0 965 40.8 96.3 4,891 
Cohort 2 / participant started 2018-2019 84 0 146 31.7 35.2 2,659 
School level 
Elementary school participants 110 0 965 46.4 101.8 5,103 
K-8 participants 56 0 111 29.0 27.0 1,624 
Middle school participants 91 0 206 28.4 37.5 2,587 
6-12 participants 32 0 732 82.1 157.1 2,627 
Content area taught 
Elementary/multiple subjects 109 0 253 40.3 49.8 4,391 
Secondary ELA 47 0 732 68.4 131.6 3,214 
Secondary social studies/history 45 0 519 41.5 81.8 1,869 
Secondary science 24 0 99 28.4 26.7 682 


Differences in module viewing by participant role were dramatic. Teacher leaders viewed 
over twice the number of modules as teachers on average (97.6 to 43.1). Administrators also 
viewed modules much less frequently (2.7). Participants who began participating in LDC in the 
same year their school launched the program viewed more modules than participants who 
joined PLCs in later years (the difference was particularly pronounced at Cohort 1 schools). 
Participants in elementary and 6-12 schools viewed more modules than their peers in middle 
and K-8 schools. Finally, secondary ELA teachers viewed a substantially greater number of 
modules on average than peers, while secondary science teachers viewed fewer modules. 
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Module Editing 


As shown in Table 4.3, on average participants who engaged in editing modules did so 16 


times over the course of the school year. There was a wide range of engagement from editing 


zero times to making 363 edits to modules. Eighty-vie participants with CoreTools user accounts 


(29%) did not do any editing of modules. 


Table 4.3 
Descriptive Statistics for the Number of Times a Participant Edited a Module in CoreTools, by 
Participant Subgroup 
Subgroup N Min Max Mean SD Sum 
All participants 289 0 363 15.9 31.2 4,604 
Participant role 
All teachers 250 0 173 16.8 24.6 4,205 
All teacher leaders 37 0 363 43.3 68.0 1,602 
All administrators 36 0 8 0.4 1.5 13 
Cohort 
Cohort 1 / teacher started 2016-2017 40 0 173 25.5 44.1 1,019 
Cohort 1 / teacher started 2017-2018 18 0 123 16.8 28.3 302 
Cohort 1 / teacher started 2018-2019 27 0 63 13.8 15.4 373 
Cohort 2 / teacher started 2017-2018 120 0 363 16.9 37.0 2,025 
Cohort 2 / teacher started 2018-2019 84 0 60 10.5 13.8 885 
School level 
Elementary school participants 110 0 363 14.4 37.8 1,580 
K-8 participants 56 0 60 13.4 14.9 752 
Middle school participants 91 0 123 15.2 20.4 1,383 
6-12 participants 32 0 173 27.8 47.4 889 
Content area taught 
Elementary/multiple subjects 109 0 84 12.8 17.5 1,400 
Secondary ELA 47 0 173 25.6 35.3 1,201 
Secondary social studies/history 45 0 173 18.7 31.5 842 
Secondary science 24 0 47 11.3 12.7 271 


Subgroup results for the module editing largely mirrored those found for page viewing. 
There was a dramatic difference between teacher leaders and teachers, with teacher leaders 
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editing more than twice the number of modules on average (43.3 compared to 16.8). Most 
administrators did not engage at this level in 2018-2019, with just three administrators making 
any edits. As with module viewing, participants who began participating in LDC in the same year 
their school launched the program edited more modules than participants who joined PLCs in 
later years. Teachers in 6-12 schools as well as those teaching secondary ELA edited modules at 
greater levels than their peers. 


Table 4.4 


Descriptive Statistics for the Number of Times a Participant Commented on a Module in 
CoreTools, by Participant Subgroup 


Subgroup N Min Max Mean SD Sum 


All participants 289 0 209 3.1 14.3 894 


Participant role 


All teachers 250 0 100 2.7 8.0 665 
All teacher leaders 37 0 209 11.7 34.8 434 
All administrators 36 0 7 0.2 1.2 8 
Cohort 
Cohort 1 / teacher started 2016-2017 40 0 100 6.7 17.5 266 
Cohort 1 / teacher started 2017-2018 18 0 22 3.4 5.2 61 
Cohort 1 / teacher started 2018-2019 27 0 5 1.3 1.5 34 
Cohort 2 / teacher started 2017-2018 120 0 209 3.6 19.5 434 
Cohort 2 / teacher started 2018-2019 84 0 9 1.2 2.0 99 
School level 
Elementary school participants 110 0 209 4.8 22.4 522 
K-8 participants 56 0 39 2.5 5.8 141 
Middle school participants 91 0 22 1.7 3.3 151 
6-12 participants 32 0 20 2.5 5.3 80 
Content area taught 
Elementary/multiple subjects 109 0 100 3.0 10.8 323 
Secondary ELA 47 0 39 4.2 7.4 195 
Secondary social studies/history 45 0 10 1.5 2.7 68 
Secondary science 24 0 5 1.3 1.5 32 
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Module Commenting 


Only 127 participants, 44% of all participants, made at least one comment on a module. As 
shown in Table 4.4, while participants commented between zero and 209 times, the average 
was just over three times. Across participant roles, teacher leaders had the highest level of 
engagement, with about 12 comments on average. As with module viewing and editing, 
participants who began participating in LDC in the same year their school launched the program 
commented on more modules than participants who joined PLCs in later years. Elementary 
level and secondary ELA participants made more comments than their peers on average. 


Figure 4.1 
Mean Number of CoreTools Views, Edits, and Comments Made by Teachers Who Did and Did Not 
Complete and Teach Module 
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m Teachers who completed and taught modules (n=83) 


m Teachers who did not complete and teach modules (n=170) 


4.3 CoreTools Engagement as an Implementation Variable 


To evaluate the validity of CoreTools engagement as an indicator of LDC implementation, 
we examined the relationship between the three CoreTools engagement measures and module 
implementation. As described in the next chapter, CRESST identified full-length modules that i3 
teachers created and/or adapted and taught in their classrooms. While all teachers may have 
taught the module(s) in their classrooms, we took the presence of uploaded student work as 
evidence of teachers having done so. Figure 4.1 displays the mean number of CoreTools views, 
edits, and comments for the 83 participants who completed and taught at least one full-length 
LDC module and their 170 peers for whom we do not have evidence of full module 
implementation. As shown, teachers who completed and taught full-length LDC modules 
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exhibited considerably more engagement with CoreTools than their peers, across all three 
metrics. The results suggest that participants who engage deeply with the module building 
platform are more likely to complete and teach LDC modules. 


4.4 Summary of Results 


While descriptive analyses of CoreTools user behavior showed evidence of broad 
engagement with LDC’s module building platform, the depth of that engagement varied greatly 
across users. Nearly all 2018-2019 LDC participants, including principals, created a CoreTools 
user account, and the majority of teachers in PLCs viewed and edited modules. The average 
teacher, however, only made 17 edits to modules within CoreTools. While this might be 
interpreted as evidence that teachers’ engagement in instructional design was somewhat 
limited, the CoreTools data do not provide any measures of instructional design work 
conducted offline by individuals or groups of teachers. 


Furthermore, this descriptive analysis provides a broad rather than deep look, and does 
not consider LDC expectations that teachers will do very little design work on their first module 
and extensive instructional design on their later modules. Teacher leaders on average engaged 
at higher levels than their peers not playing a leadership role in the LDC intervention. Teachers 
who began participating in LDC at the same time that their schools launched the program 
engaged at higher levels than teacher who joined in later years. Participants teaching secondary 
ELA also engaged at higher levels than their peers. 
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5.0 Module Artifact Analysis 


This section presents results for the analysis of modules that LDC teachers submitted 
during the 2018-2019 school year. We first present overall results across the grade bands. This 
is followed by separate results for the elementary and secondary modules. Finally, we present 
results of an exploratory analysis examining module rating results across time for teachers who 
submitted modules across the last two school years, the qualitative results, and a summary of 
the findings. Appendix F has the Generalization theory results. 


Table 5.1 shows the overall means and standard deviations by grade band for the module 
artifact analysis. It is noted that the secondary modules had similar or higher ratings than the 
elementary modules on most of the dimensions. More specifically, mean ratings for the 
secondary modules ranged from 3.02 to 3.41, indicating that on average all dimensions were 
moderately present or realized. The mean ratings for the elementary modules ranged from the 
high twos to low threes (2.88 to 3.29), with all dimensions or approaching or being moderately 
present or realized. While the highest mean for the elementary modules was for Dimension 3, 
which measures fidelity to LDC instruction, this was the lowest for the secondary modules 
(3.02). Finally, the highest mean for the secondary modules was for Dimension 4, which 
measures the quality of the instructional strategies (3.41). 


Table 5.1 


Means and Standard Deviations for the Modules 


Dimension Elementary Secondary Overall 
Dimensions n=48 n=61 n=109 
1. Effective writing task 3.12 (1.21) 3.08 (1.26) 3.10 (1.23) 
2. Standards alignment 2.88 (1.06) 3.23 (0.97) 3.07 (1.02) 
3. Fidelity to LDC instruction 3.29 (1.18) 3.02 (1.18) 3.14 (1.18) 
4. Quality instructional strategies 3.10 (1.13) 3.41 (1.16) 3.28 (1.15) 
5. Coherence/clarity of module 3.12 (1.12) 3.21 (1.03) 3.17 (1.07) 
6. Overall impression 3.10 (0.93) 3.28 (0.76) 3.20 (0.84) 
Average (Dimensions 1 to 5) 3.10 (0.94) 3.19 (0.75) 3.15 (0.84) 


Furthermore, as one would hope to find, the average summary scores for the first five 
dimensions were similar to the overall impression ratings (Dimension 6) provided by the expert 
teachers. Finally, when examining the frequencies, about 40% of the elementary and secondary 
modules, respectively, received ratings of four or five on the different dimensions. Among the 
elementary modules, the proportion was lower for Dimension 2 (25%) and was higher for 
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Dimension 3 (50%). Finally, the secondary modules had a higher proportion of ratings of four 
and five on Dimension 4 (59%). Greater detail about the frequencies can be found in 
Appendix F. 


5.1 Analysis of Elementary Modules 


The following section presents descriptive results for the elementary modules 
disaggregated by content area and then by LDC cohort. Results from the generalizability theory 
modules as well as the expanded descriptive results for the elementary modules can be found 
in Appendix F. 


Descriptive Results by Content Area 


Table 5.2 presents descriptive results for the elementary modules by content area. A few 
comparative observations can be made. First, most of the means were in the low to mid threes 
for each of the content areas. The dimensions that received means below three involved 
Dimension 1 for the ELA sample (2.75) and Dimension 2 for the science and social studies 
samples (2.54 and 2.92). Furthermore, the science modules received a mean of 4.00 on 
Dimension 3, which measures fidelity to LDC instruction. Second, the science modules received 
higher means than the other content areas on all of the dimensions except for Dimension 2. 
Third, it can be seen that the average ratings across Dimensions 1 to 5 (2.97 to 3.36) were 
similar to those provided by the expert raters for Dimension 6 (3.00 to 3.27). Finally, the 
proportion of ratings of four or greater out of a possible five were generally less than 50% for 
the different dimensions. The exceptions for the science modules involved Dimensions 1 and 3 
(64%, respectively) and for the social studies modules also involved Dimension 3 (62%). 


Table 5.2 


Means and Standard Deviations for the Elementary Modules by Content Area 


Dimensions ELA Science Social Studies 
n=24 n=11 n=13 
1. Effective writing task 2.75 (1.33) 3.73 (0.65) 3.31 (1.18) 
2. Standards alignment 3.00 (0.88) 2.54 (1.13) 2.92 (1.32) 
3. Fidelity to LDC instruction 3.00 (1.14) 4.00 (0.89) 3.23 (1.30) 
4. Quality instructional strategies 3.08 (1.25) 3.18 (0.98) 3.08 (1.12) 
5. Coherence/clarity of module 3.00 (1.18) 3.36 (1.03) 3.15 (1.14) 
6. Overall impression 3.08 (0.97) 3.27 (0.79) 3.00 (1.00) 
Average (Dimensions 1 to 5) 2.97 (0.98) 3.36 (0.64) 3.14 (1.10) 
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Descriptive Results by Cohort 


Table 5.3 presents descriptive results for the elementary modules for the three Cohort 1 
groups (2016-2017, 2017-2018, 2018-2019) and two Cohort 2 groups (2017-2018, 
2018-2019). Because of the large differences in sample sizes, we focus our discussion on the 
groups with five or greater modules. First, as can be seen, means for the Cohort 2 teachers who 
started in 2017-2018 (2.61 to 3.04) scored lower than those found for the Cohort 2 teachers 
who started in 2018-2019 (3.26 to 3.74). Second, when looking at the overall means, the two 
cohorts had different strengths. More specifically, Cohort 1 modules received the highest 
means for Dimensions 4 to 6, while Cohort 2 modules received their highest means for 
Dimensions 1 and 3. Third, when comparing the ratings for Dimension 6 with the average across 
Dimensions 1 to 5, means are very similar for Cohort 2, but are moderately high for Cohort 1. 
Finally, when examining the proportion of ratings of four or five, percentages were generally 
highest for Dimension 3 (see Appendix F). 


Table 5.3 
Means and Standard Deviations for the Elementary Modules by i3 Cohort Grouping 


Dimensions 2016-2017 2017-2018 2018-2019 Total 

COHORT 1 n=4 n=1 n=1 n=6 
1. Effective writing task -- -- -- 2.50 (0.55) 
2. Standards alignment -- -- -- 2.67 (1.03) 
3. Fidelity to LDC instruction -- -- -- 2.83 (0.98) 
4. Quality instructional strategies -- -- -- 3.17 (1.17) 
5. Coherence/clarity of module -- -- -- 3.17 (1.17) 
6. Overall impression -- -- -- 3.17 (0.75) 
Average (Dimensions 1 to 5) -- -- -- 2.87 (0.66) 

COHORT 2 NA n=23 n=19 n=42 
1. Effective writing task NA 2.96 (1.06) 3.53 (1.43) 3.21 (1.26) 
2. Standards alignment NA 2.61 (1.08) 3.26 (0.99) 2.90 (1.08) 
3. Fidelity to LDC instruction NA 3.04 (1.40) 3.74 (0.80) 3.36 (1.21) 
4. Quality instructional strategies NA 2.83 (1.23) 3.42 (0.96) 3.10 (1.14) 
5. Coherence/clarity of module NA 2.87 (1.29) 3.42 (0.840 3.12 (1.13) 
6. Overall impression NA 2.78 (1.04) 3.47 (0.70) 3.10 (0.96) 
Average (Dimensions 1 to 5) NA 2.86 (1.08) 3.47 (0.73) 3.14 (0.98) 


Note. Means not presented for samples of less than five. 
Members of Cohort 2 did not participate in LDC during the 2016-2017 school year. 
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5.2 Analysis of Secondary Modules 


The following section presents descriptive results for the secondary modules 
disaggregated by content area and then by cohort group. Results from the generalizability 
theory modules as well as the expanded descriptive results for the secondary modules can be 
found in Appendix F. 


Descriptive Results by Content Area 


Table 5.4 presents descriptive results for the secondary modules by content area. A few 
comparative observations can be made. First, the ELA modules generally had higher mean 
ratings than the ones in the other two content areas. For example, the ELA means ranged from 
2.97 to 3.74, indicating that the dimensions were moderately present or realized. In contrast, 
the science means ranged from 2.80 to 3.50 and the social studies means ranged from 2.40 to 
3.20. Second, while the science modules received the highest mean ratings for the first three 
dimensions, which focused on foundational issues, the ELA and studies modules had higher 
means for the last three dimensions. Third, when comparing the average ratings across 
Dimensions 1 to 5 with the expert ratings for Dimension 6, the results showed small 
differences. Means for Dimension 6 were slightly higher for ELA and social studies, and were 
slightly low for science. Finally, when looking at the proportion of ratings of four or greater out 
of a possible five, percentages were fairly large for ELA and science (42% to 77%, 40% to 60%), 
but were much lower for social studies (15% to 40%). 


Table 5.4 


Means and Standard Deviations for Secondary Modules by Content Area 


Dimensions ELA Science Social Studies 
n=31 n=10 n=20 
1. Effective writing task 3.39 (1.02) 3.50 (1.27) 2.40 (1.35) 
2. Standards alignment 3.42 (0.85) 3.30 (1.49) 2.90 (0.79) 
3. Fidelity to LDC instruction 2.97 (1.38) 3.30 (1.16) 2.95 (0.82) 
4. Quality instructional strategies 3.74 (1.15) 2.80 (1.32) 3.20 (0.95) 
5. Coherence/clarity of module 3.48 (0.85) 2.80 (1.32) 3.00 (1.08) 
6. Overall impression 3.52 (0.68) 3.00 (1.15) 3.05 (0.51) 
Average (Dimensions 1 to 5) 3.40 (0.68) 3.14 (1.15) 2.89 (0.50) 
Results by Cohort 


Table 5.5 presents descriptive results for the secondary modules for the three Cohort 1 
groups (2016-2017, 2017-2018, 2018-2019) and two Cohort 2 groups (2017-2018, 
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2018-2019). First, most of the mean ratings were in the low to mid threes. The exceptions for 
Cohort 1 included Dimension 1 for those who started in 2016-2017, and Dimensions 1, 2, and 5 
for those who started LDC in 2017-2018. The only exception for Cohort 2 includes Dimension 3 
for those who started in 2018-2019. Modules for the two Cohorts also have different strengths 
and weaknesses. More specifically, the highest means for Cohort 1 were for Dimension 4, which 
measures quality instructional strategies and were lowest for Dimension 1, which measures the 
effective writing task. In contrast, for Cohort 2, the lowest means were for Dimension 3, which 
measures fidelity to LDC instruction. The highest means for Cohort 2 were for Dimension 1 
(2017-2018) and Dimension 2 (2018-2019). When comparing results for Dimension 6 with the 
averages for Dimensions 1 to 5, means generally showed small to very small differences. Finally, 
when looking at the proportion of ratings of four or greater out of a possible five, percentages 
were generally over 50% for the Cohort 2 teachers who started LDC in 2017-2018. In contrast, 
percentages were often below 50% for the other cohort groups (see Appendix F). 


Table 5.5 


Means and Standard Deviations for the Secondary Modules by i3 Cohort Grouping 


Dimensions 2016-2017 2017-2018 2018-2019 Total 

COHORT 1 n=14 n=14 n=9 n=34 
1. Effective writing task 2.93 (1.14) 2.54 (1.63) 3.00 (1.41) 2.82 (1.36) 
2. Standards alignment 3.29 (0.91) 2.73 (0.79) 3.44 (1.01) 3.15 (0.92) 
3. Fidelity to LDC instruction 3.00 (1.18) 3.09 (0.83) 3.11 (1.17) 3.06 (1.04) 
4. Quality instructional strategies 3.57 (1.34) 3.36 (0.92) 3.47 (1.28) 3.47 (1.16) 
5. Coherence/clarity of module 3.50 (1.16) 2.82 (1.08) 3.00 (1.00) 3.15 (1.10) 
6. Overall impression 3.43 (0.76) 3.00 (0.63) 3.11 (0.78) 3.21 (0.73) 
Average (Dimensions 1 to 5) 3.26 (0.77) 2.91 (0.73) 3.20 (0.72) 3.13 (0.74) 

COHORT 2 NA n=17 n=10 n=27 
1. Effective writing task NA 3.65 (0.93) 3.00 (1.15) 3.41 (1.05) 
2. Standards alignment NA 3.41 (1.06) 3.20 (1.03) 3.33 (1.04) 
3. Fidelity to LDC instruction NA 3.24 (1.44) 2.50 (1.08) 2.96 (1.34) 
4. Quality instructional strategies NA 3.47 (1.28) 3.10 (0.99) 3.33 (1.18) 
5. Coherence/clarity of module NA 3.47 (0.87) 3.00 (1.05) 3.30 (0.95) 
6. Overall impression NA 3.59 (0.80) 3.00 (0.67) 3.37 (0.79) 
Average (Dimensions 1 to 5) NA 3.45 (0.77) 2.96 (0.69) 3.27 (0.77) 


Note. Members of Cohort 2 did not participate in LDC during the 2016-2017 school year. 
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5.3 Exploratory Analysis of Modules 


As previously noted, in order to examine potential growth among returning teachers, 
ratings were compared for teachers who were the first author and submitted complete 
modules in both the 2017-2018 and 2018—2019 school years. The following presents 
descriptive results for the last complete module submitted by these teachers. Tables 5.6 and 
5.7 present results for the 13 elementary teachers and 30 secondary teachers who submitted 
complete modules as specified during both school years. 


When examining the elementary results, change in performance varied by dimension (see 
Tables 5.6 and 5.7). More specifically, means increased slightly (0.08 to 0.15) for Dimensions 3 
to 5, which measure fidelity to LDC, quality instructional strategies, and coherence and clarity. 
In addition, more than half of the elementary teachers showed no change or had improved 
ratings for these three dimensions (53.8% to 76.9%). In contrast, means dropped by about one- 
half point for Dimension 6 across the two school years, and more than half of the teachers 
showed a drop in their rating (53.8%). Finally, means dropped by approximately one point for 
Dimensions 1 and 2 across the two school years. In this case, no teachers showed an 
improvement for Dimension 1 and only one showed an improvement across the 2 years for 
Dimension 2. 


When examining the secondary results, means were similar across the two school years 
for most of the dimensions. For example, means increased by 0.03 for Dimensions 2 and 6 and 
went down by only 0.10 to 0.13 for Dimensions 1, 4, and 5. The only dimension with a 
moderate decrease (0.33) measured fidelity to LDC instruction (Dimension 3). Furthermore, 
one-third or more of teachers had an increase in ratings for all of the dimensions except 
Dimension 3, which measures fidelity to LDC instruction. Moderate percentages of teachers 
also showed no change across the 2 years for Dimensions 2 to 6 (26.7% to 40.0%). 


We also calculated cohort level results for the elementary and secondary teachers. As 
with the overall elementary results, the Cohort 2 teachers showed mixed results by dimension, 
with moderate to large decreases (0.56 and 0.89) for the first two dimensions, and a large 
increase of 0.78 for Dimension 3. In contrast, for the secondary modules the Cohort 1 means 
showed small to moderate decreases (0.10 to 0.40) and the Cohort 2 means showed small to 
large increases for all dimensions except Dimension 3. Finally, while half of the secondary 
Cohort 2 teachers showed decreases for Dimension 3, which measures fidelity to LDC 
instruction, the other half showed no change or improvement. Despite this, these results 
should be considered tentative because of the large differences in sample size between the 
cohorts. See Appendix F for further results. 
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Table 5.6 


Means and Standard Deviations for the Exploratory Analysis of Modules 


Dimension 2017-2018 2018-2019 Change 
ELEMENTARY (n = 13) 
1. Effective writing task 4.15 (0.90) 3.23 (0.83) -0.92 (0.86) 
2. Standards alignment 3.92 (0.95) 2.85 (0.99) -1.08 (1.04) 
3. Fidelity to LDC instruction 3.38 (1.32) 3.54 (1.27) 0.15 (1.91) 
4. Quality instructional strategies 3.31 (0.75) 3.46 (1.20) 0.15 (1.40) 
5. Coherence/clarity of module 3.46 (0.78) 3.54 (1.27) 0.08 (1.19) 
6. Overall impression 3.77 (0.60) 3.31 (0.86) -0.46 (0.88) 
Average (Dimensions 1 to 5) 3.65 (0.58) 3.32 (0.91) -0.32 (0.90) 
SECONDARY (n = 30) 
1. Effective writing task 3.27 (1.23) 3.13 (1.31) -0.13 (1.55) 
2. Standards alignment 3.07 (1.08) 3.10 (0.92) 0.03 (1.47) 
3. Fidelity to LDC instruction 3.50 (1.25) 3.17 (1.18) -0.33 (1.27) 
4. Quality instructional strategies 3.70 (1.02) 3.60 (1.19) -0.10 (1.60) 
5. Coherence/clarity of module 3.50 (0.82) 3.37 (1.07) -0.13 (1.25) 
6. Overall impression 3.40 (0.86) 3.43 (0.73) 0.03 (1.27) 
Average (Dimensions 1 to 5) 3.41 (0.75) 3.27 (0.78) -0.13 (1.13) 
OVERALL (n = 43) 
1. Effective writing task 3.53 (1.20) 3.16 (1.17) -0.37 (1.41) 
2. Standards alignment 3.33 (1.11) 3.02 (0.94) -0.30 (1.44) 
3. Fidelity to LDC instruction 3.47 (1.26) 3.28 (1.20) -0.19 (1.48) 
4. Quality instructional strategies 3.58 (0.96) 3.56 (1.18) -0.02 (1.54) 
5. Coherence/clarity of module 3.49 (0.80) 3.42 (1.12) -0.07 (1.22) 
6. Overall impression 3.51 (0.80) 3.40 (0.76) -0.12 (1.18) 
Average (Dimensions 1 to 5) 3.48 (0.71) 3.29 (0.81) -0.19 (1.06) 
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Table 5.7 


Counts and Percentages for the Exploratory Analysis of Modules 


Negative No Change Positive 

Dimension # % # % # % 
ELEMENTARY (n = 13) 
1. Effective writing task 8 61.5 5 38.5 0 0.0 
2. Standards alignment 10 76.9 2 15.4 1 7.7 
3. Fidelity to LDC instruction 6 46.2 2 15.4 5 38.5 
4. Quality instructional strategies 3 23.1 4 30.8 6 46.2 
5. Coherence/clarity of module 5 38.5 2 15.4 6 46.2 
6. Overall impression 7 53.8 4 30.8 2 15.4 
Average (Dimensions 1 to 5) 7 53.8 2 15.4 4 30.8 
SECONDARY (n = 30) 
1. Effective writing task 13 43.3 5 16.7 12 40.0 
2. Standards alignment 10 33.3 10 33.3 10 33.3 
3. Fidelity to LDC instruction 14 46.7 11 36.7 5 16.7 
4. Quality instructional strategies 8 26.7 12 40.0 10 33.3 
5. Coherence/clarity of module 12 40.0 8 26.7 10 33.3 
6. Overall impression 9 30.0 11 36.7 10 33.3 
Average (Dimensions 1 to 5) 15 50.0 3 10.0 12 40.0 
OVERALL (n = 43) 
1. Effective writing task 21 48.8 10 23.2 12 27.9 
2. Standards alignment 20 69.8 12 27.9 11 25.6 
3. Fidelity to LDC instruction 20 69.8 13 30.2 10 23.2 
4. Quality instructional strategies 11 25.6 16 37.2 16 37.2 
5. Coherence/clarity of module 17 39.5 10 23.2 16 37.2 
6. Overall impression 16 37.2 15 34.9 12 27.9 
Average (Dimensions 1 to 5) 22 51.2 5 11.6 16 37.2 


5.4 Qualitative Results 


Expert raters were asked to write notes about the reasoning behind their ratings for each 
module. This was done to assist in the consensus making process for all modules coded by more 
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than one expert teacher, to shed light on the ratings in general, and to suggest potential ways 
to improve ratings in the future. The following presents key themes that emerged concerning 
Dimensions 1 through 5, as well as our expert raters’ overall impression. 


Dimension 1: Effective Writing Task 


While more than three-fourths of the modules received ratings of three to five on this 
dimension, the writing task was not always realized. Modules that received low ratings tended 
to have less information for the expert raters to evaluate. For example, many modules lacked a 
background for students, an extension activity, and/or summary information on the coversheet. 
When provided, these components were not always clear and, in some cases, did not match the 
rest of the module. 


Dimension 2: Standards Alignment 


While standards alignment was generally good, higher ratings of four or five were much 
more common for the secondary modules than for the elementary ones. When providing lower 
ratings for this dimension, the expert raters noted the following issues. First, modules tended to 
have reading standards, but were less likely to have key writing, science, or social studies 
standards that matched the task and instruction. Second, some of the modules had standards 
at different grade bands—primary and upper elementary or elementary and secondary—but 
did not always provide an explanation for this misalignment (e.g., special needs classes, 
remedial coursework, etc.). Third, some modules included standards from other states. 


Dimension 3: Fidelity to LDC Module Instruction 


One of the goals of Dimension 3 is to examine the distribution of activities and time spent 
on the four stages of the LDC instructional practice (preparation for the task, reading process, 
transition to writing, and writing process). Modules that received low ratings tended to be 
missing skills and mini tasks that our expert teachers thought were necessary to meet the 
writing task and standards. In some cases, modules had skills listed, but did not provide a mini 
task, or the mini task(s) provided had scant information about the planned instruction. Low 
rated modules also lacked additional information in the module summary, background, 
extension, and/or reflection from which to judge the reason(s) for the omissions. 


Dimension 4: Quality Instructional Strategies 


Modules that received high ratings of four or five for this dimension were noted to be 
thorough and detailed, and were sometimes noted to have student work that reflected this 
quality. In contrast, modules that received low ratings of one or two often had minimal details 
and/or student work that did not reflect the strategies. In some instances, expert teachers also 
noted that instructional strategies did not match the task, were not specific to the lesson, 
and/or did not match the grade level or grade band specified. Finally, raters noted that it was 
more difficult to determine whether the strategies were realized when only one student work 
sample was attached, or the attachments only reflected one level (e.g., only advanced). 
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Dimension 5: Coherence and Clarity of Module 


Modules that received low ratings of one or two for Dimension 5 tended to have 
weaknesses across the other dimensions and components. For example, expert raters tended 
to notice issues with the quality of the writing task, standards alignment, skills, and 
instructional strategies. Raters also noted issues with student work not matching other 
components, or providing limited evidence for the goals of the lesson being met. 


Dimension 6: Overall Impressions 


Modules that received the highest rating of five for Dimension 6 were frequently 
described in very positive terms. Expert raters called these modules strong, focused, great, well 
planned, and thorough. In addition, during debriefings expert raters talked about how these 
modules were ready to implement. In contrast, modules that received low ratings of one or two 
tended to receiving comments about being incomplete or missing many or most of the 
expected components. In some cases, comments started out stronger, but raters found that the 
skills, instruction, and/or student work did not realize the potentials of the module. 


5.5 Summary of Results 


Based on the primary analyses, teachers as a whole did not show growth in the quality of 
their modules over the last two school years. More specifically, average quality for the 
elementary modules went down by a half point (3.57 to 3.10) and the quality of the secondary 
modules went down slightly (3.34 to 3.19). As such, the average module was moderately 
present or realized during both school years. Despite this, moderate proportions of elementary 
(25% to 50%) and secondary modules (38% to 59%) received ratings of four or five on the 
dimensions, representing features that are sufficiently to fully present or realized. 


Differences in quality were also found when examining results by subgroup. For example, 
quality for the elementary modules was strongest for science and for Cohort 2 teachers who 
started during the 2018-2019 school year, and was weakest for ELA and Cohort 2 teachers who 
started during the 2017-2018 school year. In contrast, differences were found by dimension for 
the secondary modules. More specifically, means for Dimension 1, which evaluates the writing 
task, were highest for the science modules and those created by Cohort 2 teachers who begin 
implementing LDC during 2017-2018. Furthermore, means for Dimensions 4 and 5, which 
evaluate the instructional strategies as well as coherence and clarity, were highest for 
secondary ELA and modules developed by Cohort 1 teachers who began LDC during 2016-2017. 
Despite this, because of the variation in sample sizes, these differences should be considered 
tentative. 


An exploratory analysis was also conducted to examine growth in performance for the 13 
elementary teachers and 30 secondary teachers who submitted complete modules with 
student work during both the 2017-2018 and 2018-2019 school years. As with the primary 
analysis, the longitudinal analysis showed somewhat different results for the two grade band 
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samples. More specifically, module quality for the elementary teachers went down by 
approximately one point (0.92 and 1.08) for Dimensions 1 and 2, and dropped approximately 
one-half point (0.46) on Dimension 6. Module quality for the elementary teachers also 
increased slightly (0.08 to 0.15) for Dimensions 3 to 5. In contrast, overall results were generally 
flat for the secondary modules. In this case, a moderate decrease in quality was found for 
Dimension 3 (0.33), and very small decreases in secondary module quality were found for 
Dimensions 1, 4, and 5. In addition, the proportion of modules that showed no change or 
increased in quality was typically greater for the secondary modules (53.3% to 73.3%) than for 
the elementary modules (23.1% to 76.9%) on the different dimensions. Finally, some 
differences were found by cohort for the secondary teachers, with those in Cohort 1 showing 
decreases in module quality across dimensions and those in Cohort 2 showing increases in 
quality for all dimensions except Dimension 3. 
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6.0 Fidelity of Implementation Analysis 


This chapter describes the results of a broad analysis of the LDC schools’ fidelity of 
implementation to the LDC model. As part of Department of Education’s requirements, UCLA 
CRESST and LDC collaborated to create a fidelity matrix to measure schools’ fidelity of 
implementation across four key components of the LDC model. Within the matrix, each key 
component is measured via multiple indicators, with 14 indicators total across the key 
components. The key components, as depicted in the LDC logic model (see Figure 1.1) and 
described in Chapter 1 are (a) common planning time for LDC professional learning community 
with synchronous coach support; (b) asynchronous support from LDC coaches; (c) teacher 
implementation activities; and (d) leadership support. 


The fidelity matrix can be found in Appendix G. The matrix outlines a process whereby 
fidelity scores on each indicator are computed for each school, and for the program as a whole. 
The matrix also produces a score for whether the program met fidelity for each of the four 
components. Although the process produces school level fidelity scores for all indicators, the 
initial unit of implementation is either teacher, module, or school depending on the indicator. 
LDC and CRESST staff worked collaboratively to construct the fidelity matrix, with LDC staff 
setting the thresholds for adequate implementation on each indicator and at each level of 
implementation (teacher/module, school, and program). 


Data for these 14 fidelity indicators come from four sources. Three indicators are based 
on data collected via CRESST’s teacher survey. Data on these selected survey items are used to 
produce school- and program-level fidelity scores. Three indicators are based on data from the 
PLC reflection form collected by LDC, which was filled out by teacher leaders at the school level 
and used to capture information on both full PLC sessions and one-on-one planning and 
progress calls between teacher leaders and coaches. Two indicators are based on LDC 
administrative records capturing attendance by teacher leaders and administrators at in-person 
meetings occurring four times per year. Finally, six indicators are based on LDC’s CoreTools 
analytic data capturing a variety of behaviors including teachers’ viewing of LDC online course 
content in the LEARN portion of CoreTools, editing of modules, and uploading of student work; 
and coaches’ commenting and providing peer review on modules. This report explores and 
presents the overall amount of editing and commenting behavior of teacher participants in 
Chapter 4. Here we focus on whether participants edited key portions of the module, as well as 
the comments provided by coaches on modules as a measure of their asynchronous feedback. 


In this chapter, we first present the overall results for the school- and program-level 
fidelity scores for all the indicators. We then provide more detailed descriptive results with 
sections for each of the four key components and subsections for each of the 14 fidelity 
metrics. Next, we compare fidelity of implementation results to the prior year (2017-2018). 
We then explore the relationship between the fidelity of implementation indicators and the 
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student ELA outcome scores at both the school and teacher level in 2018-2019. Finally, we 
summarize results. 


6.1 School- and Program-Level Fidelity of Implementation Scores 


Table 6.1 provides a summary for how LDC schools performed across the four key 
components and 14 indicators of fidelity of implementation. As described in detail in the fidelity 
matrix in Appendix G, schools were rated on 4-point scales (from 0 to 3) on each indicator. At 
the school level, adequate implementation was defined as a score of at least 2. For many of the 
indicators, the 4-point scale is based on the proportion of teachers who met a certain 
implementation threshold. In order for the program as a whole to meet fidelity on a particular 
indicator, a designated proportion of schools had to meet the school-level threshold of 2. For 
example, program-level fidelity for the Key Component 2 indicators were each met if half or 
more of the schools met fidelity. For all the other indicators (under Key Components 1, 3, and 
4) program-level fidelity was met if three quarters of schools met fidelity. 


As displayed in Figure 6.1, there was great variation across schools in their success in 
meeting the implementation thresholds. Schools met fidelity on a range of between 0 and 8 
indicators. On average schools met fidelity on 4.4 indicators. Furthermore, no schools met 
fidelity on more than eight indicators, which suggests that all participating schools had 
substantial room for growth. 


Figure 6.1. 
Frequency Distribution of Schools Reaching Adequate Implementation on Different Numbers of Fidelity 
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Table 6.1 


Summary Table of School- and Program-Level Fidelity Scores by Indicator in 2018-2019 


% of schools 


Program 
Level Level Level Level fi en 4 
Key component Indicator 0 1 2 3 ers 
Key component Teacher attendance at weekly 50 14 6 14 No 
1: Common PLC meetings 
plcnnine BE 10) Amount of time spent on LDC 17 61 17 6 No 
LDC professional : pees 
‘ during common planning time 
learning 
community with Exposure to LDC LEARN 86 8 3 3 No 
synchronous content during first 
coach support instructional cycle 
Exposure to LDC LEARN 97 0 0 3 No 
content during second 
instructional cycle 
Perceived effectiveness of 25 6 25 42 No 
engagement in PLC on teacher 
competencies 
Key component Coach comments on modules 14 14 11 56 Yes 
2: Asynchronous 
support from Coach formative peer review 92 3 3 3 No 
LDC coaches on modules 
Teacher perception of the 25 8 22 42 Yes 
helpfulness of coach written 
feedback on modules 
Key component Module editing 64 8 11 17 No 
3: Teacher 
implementation odule implementation 94 3 3 0 No 
activities 
Key component Frequency of coach/teacher 3 42 33 22 No 
4: leadership leader monthly meetings 
Support at Administrator attendance at 36 44 14 6 No 
different levels ’ 
quarterly in-person 
administrator meetings 
Teacher leader attendance at 25 33 17 25 No 
quarterly in-person teacher 
leader meetings 
Principal mini-task 50 3 17 28 No 


observation 
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For fidelity to be met on a key component, the fidelity matrix requires that fidelity be met 
on each of the indicators for that key component. As a result, the analysis for 2018-2019 
concludes that fidelity was not met for the four key components across the LDC schools. 


Examining the fidelity scores by indicator, we found that adequate implementation at the 
program level was met on just 2 of the 14 indicators: coach comments on modules teacher 
perception of the helpfulness of coach written feedback on modules. A majority of the schools 
also met the threshold for perceived effectiveness of engagement in PLC on teacher 
competencies and frequency of coach/teacher leader monthly meetings, although those 
proportions were not sufficient for the program to be labeled as having adequate 
implementation on those indicators. For all the other indicators, a minority of the schools met 
the threshold. 


6.2 Key Component 1: Common Planning Time for LDC Professional Learning 
Community with Synchronous Coach Support 


Schools varied greatly in their level of fidelity on the five indicators of Key Component 1, 
as shown in Figure 6.2. Schools were most successful with regard to teachers’ perceptions of 
the impact of PLC engagement on their skills (perceived effectiveness of engagement in PLC on 
teacher competencies), with 24 out of 36 (67%) of the schools meeting adequate 
implementation on the indicator. A minority of the schools met fidelity on teacher attendance 
at weekly PLC meetings (19%) and amount of time spent on LDC during common planning time 
(22%). Meeting fidelity on the indicators of exposure to LDC LEARN content was challenging for 
most schools, with only a few schools reaching the adequate implementation threshold for one 
of the two metrics. 


Teacher Attendance at Weekly PLC Meetings 


PLC reflection data show that schools varied greatly in terms of the number of times their 
PLCs met over the course of the 2018-2019 school year. The number of sessions ranged from 
six to 37 times, with PLCs on average meeting 12.9 times. This is important context for the 
interpretation of the teacher attendance at weekly PLC meetings fidelity indicator. The indicator 
measures the proportion of teachers who regularly attended sessions, but does not incorporate 
the number of times that the PLC met across the year. 


Attendance rates across the full population of PLC participants varied a great deal, ranging 
from zero to 100% and averaging 74%. Classroom teachers playing an LDC teacher leader role 
(n = 28, 87%) attended at higher levels than did classroom teachers not playing a leadership 
role (n = 187, 72%). Coaches, coordinators, and administrators playing the teacher leader role 
(n = 4) attended at lower rates than classroom teachers playing the same role. The adequate 
implementation threshold for individual teacher attendance was 80% or greater, and 115 out of 
219 PLC participants (53%) reached this threshold. Only 19% of schools met the threshold of 
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75% of their PLC participants attending 80% of sessions. A majority of the schools were either at 
the lowest fidelity level or were missing data as a result of attendance data not being recorded 


properly. 


Figure 6.2 
Number of Schools Reaching Different Fidelity Levels for Each Key Component 1 Indicator 
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Amount of Time Spent on LDC During Common Planning Time 


This metric is based on PLC reflection data for the subset of PLC sessions that coaches 
joined either digitally or in person. In cases where PLC reflection data were not reported 
accurately, the typical session length was gathered via LDC staff conversations with coaches. 


School-level scores were based on the modal (most common) response on the PLC 
reflection form. For six of the schools that response was under 45 minutes, for 22 schools it was 
45-59 minutes, for 6 schools the most common response was 60-74 minutes, and two schools 
met most frequently for 75-90 minutes. Because of the nature of the data, we cannot ascertain 
the exact length of the sessions. As such it is possible that many of the sessions recorded as 45— 
59 minutes came quite close to the desired 60 minutes of PLC time. Nevertheless, according to 
the thresholds set, a majority of the schools did not meet fidelity on this indicator. 
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Exposure to LDC LEARN Content During First and Second Instructional Cycles 


Here we report on the two indicators measuring PLC participants’ engagement with the 
LEARN online course content (instructional courses that help LDC participants learn how to 
navigate an LDC instructional cycle). Almost three quarters (74%) teachers and teacher leaders 
participating in LDC PLCs in 2018-2019 viewed at least one session in each of two instructional 
cycles. Overall the full population of teachers and teacher leaders on average were exposed to 
42% of the LEARN sessions in the first instructional cycle and 28% of the LEARN sessions in the 
second instructional cycle. This represents a marked improvement over the prior year, when 
participants on average were exposed to 9% of LEARN sessions in the first instructional cycle 
and 5% in the second instructional cycle. Despite this improvement in the average exposure, 
the great majority of teachers and teacher leaders were still not meeting the fidelity threshold 
of exposure to 60% of instructional cycle sessions. Only 29% of participants met this threshold 
for the first instructional cycle, and 15% of participants met it for the second instructional cycle. 


Overall a total of two schools met the adequate implementation threshold for the first 
instructional cycle (scoring at Fidelity Levels 2 and 3) and one school met the threshold for the 
second instructional cycle (the two groups of schools did not overlap). Nearly all the remaining 
schools scored at the lowest fidelity score level. 


Perceived Effectiveness of Engagement in PLC on Teacher Competencies 


This indicator is based on the 183 teachers who provided data on Question 30 in the 
teacher survey (see Appendix A), which asks teachers to report how much their skills had 
improved in a number of areas aligned with LDC core competencies. Here we base fidelity 
levels on an index averaging the seven survey items (see Question 30 in Appendix A), which are 
each measured ona scale of 1 to 4. The threshold for adequate implementation at the teacher 
level is an index score of 3, corresponding to the survey response reporting moderate 
improvement in the skill area. Across the whole sample, 9 teachers (5%) had scores of 1 to less 
than 2 (no to a little improvement), 27 (16%) had scores of 2 to less than 3 (a little to moderate 
improvement), 83 (48%) had scores of 3 to less than 4 (moderate to great improvement), and 
53 (31%) had scores of 4 (great deal of improvement). Over three quarters of respondents in 
the sample reached the adequate implementation threshold score of 3 or greater on the index. 


There was a great deal of variation in survey responses both within and across schools. In 
seven schools, half or fewer of the teachers reported a moderate effect on the competencies. 
On the other hand, 100% of respondents in 14 schools reported a moderate or greater effect. 
The variation can be clearly seen in Figure 6.2. Despite nearly 80% of teachers reporting that 
LDC had an impact on their competencies, only two thirds of the schools had three quarters or 
more of their teachers report a moderate impact. 
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6.3 Key Component 2: Asynchronous Support from LDC Coaches 


As with Key Component 1, there was a great deal of variation across schools in their level 
of fidelity on the three Key Component 2 indicators, as shown in Figure 6.3. Nevertheless, a 
majority of schools met the adequate implementation thresholds for coach comments on 
modules and teacher perception of the helpfulness of coach written feedback on modules, and 
as a result the program met fidelity on these two metrics. Coach peer review was a lightly used 
tool, with almost all of the schools scoring at the lowest fidelity level. Together, the findings 
suggest that the program was successful with regard to coaches providing useful feedback to 
LDC participants, although expectations were not being met for the use of the more 
comprehensive peer review tool as a formative feedback mechanism. 


Figure 6.3 
Number of Schools Reaching Different Fidelity Levels for Each Key Component 2 Indicator 
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Coach Comments on Modules 


Results on this indicator are based on the universe of modules linked to a LEARN 
instructional cycle. In total there were 170 modules linked to courses across 34 of the 36 
schools, with the number of linked modules per school ranging from 1 to 14 and averaging 5. 
Those modules in turn received between 0 and 23 comments from coaches (10 modules 
received 0 comments) and an average of 5.6 comments. An adequate level of feedback on a 
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linked module was defined as having received two or more coach comments, and 140 out of 
the total 170 modules (82%) met that threshold. The proportion of modules within schools that 
met the two-comment threshold varied a great deal across schools, which accounts for the 
variation in school level fidelity scores as seen in Figure 6.3. 


Coach Formative Peer Review on Modules 


Results on this indicator are based on the universe of teachers and teacher leaders who 
co-authored at least one linked course. That population includes 148 participants across 34 
schools (two schools did not have any teachers with modules linked to courses). At the teacher 
level, fidelity was defined as having received at least one peer review from a coach. Only 58 
teachers received at least one peer review from their coach, and therefore only 6% of the 
schools met the school-level fidelity threshold. 


Teacher Perception of Helpfulness of Coach Written Feedback on Modules 


Across 175 teacher survey respondents, 138 teachers (79%) reported using written 
feedback on modules and finding it moderately to very helpful. Across schools, the proportion 
of teachers who reported at least moderate usefulness ranged from 0 to 100% and a mean of 
75% (data was not available for one school as a result of a 0% survey response rate at that 
location). Almost two thirds of schools met the adequate implementation threshold, and 
therefore the program as a whole met fidelity on this metric. 


6.4 Key Component 3: Teacher Implementation Activities 


Here we report on indicators related to teachers’ engagement with the process of 
designing and implementing LDC modules. As shown in Figure 6.4, schools were more 
successful in terms of meeting thresholds for the proportion of their teachers who engaged 
deeply in the design process by editing key portions of modules, with over a quarter of schools 
meeting fidelity on this indicator. In contrast, only one of the schools had three quarters or 
more of their teachers implement two modules, as measured by the uploading of student work 
samples. 


Module Editing 


As outlined in the fidelity matrix, teacher fidelity levels are built in a stepladder fashion. 
Teachers meeting Fidelity Level 1 were required to edit at least one task in a module. Teachers 
meeting Fidelity Level 2 (adequate implementation) were required to meet Fidelity Level 1 and 
edit either standards or texts in at least one module; teachers at this level did not edit either 
skills/mini-tasks or rubrics. Finally, teachers meeting Fidelity Level 3 met the previous two 
requirements (editing at least one task and standards or texts) and also edited skills/mini-tasks 
and/or the rubric. 
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Figure 6.4 
Number of Schools Reaching Different Fidelity Levels for Each Key Component 3 Indicator 
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A total of 119 participants (47%) failed to meet the threshold of editing the teaching task 
in one module and therefore implemented at a fidelity level of zero. Nineteen participants (7%) 
edited a task, but did not edit standards or texts, and therefore were at Fidelity Level 1. A total 
of 118 participants (46%) met the adequate implementation threshold, with 34 (13%) scoring at 
Level 2 (edited standards or texts) and 84 (33%) scoring at Level 3 (edited standards or texts 
and skills/mini-tasks or rubric). As a result, overall less than half of participants met the teacher- 
level adequate implementation threshold, and less than one third of schools met the school- 
level threshold for adequate implementation. It is important to note that in response to 
teachers collaborating on and implementing common modules in schools, LDC refined its data 
collection in the summer of 2018 to include tracking engagement data on modules that 
teachers collaborated on. The fidelity matrix, and CRESST’s analyses of editing data, were not 
designed to capture this shift toward a collaborative model of instructional design in CoreTools, 
and therefore may not fully capture engagement in the design process. 


Module Implementation 


For this indicator, our sample includes the 250 classroom teachers who participated in 
LDC in 2018-2019 (as the metric involves classroom implementation, out-of-classroom staff are 
excluded). In this case, the uploading of student work served as a proxy for whether the teacher 
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implemented a module in their classroom. The number of modules with uploaded student work 
ranged from 0 to 5 with a mean of about 0.5. One hundred sixty-seven teachers (67%) didn’t 
upload student work to any modules. Forty (16%) uploaded student work to one module, and 
38 (15%) uploaded to two modules. Five teachers (2%) uploaded student work to three or more 
modules. Overall, just 43 teachers (17%) met the adequate implementation threshold, and as a 
result only one school met fidelity on the indicator. 


6.5 Key Component 4: Leadership Support at Different Levels 


As depicted in Figure 6.5, there was a great deal of variation across schools on each of the 
Key Component 4 metrics. Depending on the metric, between 19% and 56% of schools met 
fidelity. These results were not positive enough for fidelity to be met at the program level on 
any of the metrics. 


Figure 6.5 
Number of Schools Reaching Different Fidelity Levels for Each Key Component 3 Indicator 
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Frequency of Coach/Teacher Leader Monthly Meetings 


Teacher leaders reported meeting with their coach between 3 and 21 times with a mean 
of 10 times, according to PLC reflection data and communication with coaches. The adequate 
implementation threshold was set at nine or more coach/teacher leader meetings across the 
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school year, and 20 of 36 schools (56%) met that threshold. Fifteen of 36 schools (42%) met 
between four and eight times (low implementation) and one school (3%) met less than four 
times (very low implementation). 


Administrator Attendance at In-Person Administrator Meetings 


Administrators had the opportunity to attend three administrator training events. 
Administrators in 2 schools (6%) attended all three events, in 5 schools (14%) attended two 
events, in 16 schools (44%) attended one event, and in 13 schools (36%) administrators 
attended zero events. With the adequate implementation threshold set at two of three events, 
7 out of 36 schools (19%) met the threshold. 


Teacher Leader Attendance at Quarterly In-Person Teacher Leader Meetings 


Teacher leaders also had the opportunity to attend three teacher leader training events. 
Teacher leaders attended all three events in 9 schools (35%), attended two events in 6 schools 
(17%), attended one event in 12 schools (33%), and attended zero events in 9 schools (25%). 
With adequate implementation again set at two of three events, 15 out of 36 schools (42%) 
met the threshold. 


Principal Mini-Task Observation 


Teachers were asked to report how many times their administrator observed their 
teaching of an LDC mini-task. Thirty-eight percent of survey respondents reported never having 
been observed by an administrator when teaching an LDC mini-task, 24% reported being 
observed once, 22% reported being observed twice, and 16% reported being observed three or 
more times. Under the fidelity matrix, 75% of teachers had to be observed for the school to 
meet fidelity, and only 44% of schools met this threshold. 


6.6 Change in Fidelity of Implementation From 2017-2018 to 2018-2019 


Here we explore change over time in how the program performed on the 14 different 
fidelity metrics. Table 6.2 displays the proportion of schools that met fidelity thresholds in 
2017-2018 and 2018-2019, and whether the program as a whole met fidelity on the metric in 
each year. It is important to note that the 2 years of analysis are based on two different 
samples of schools. Seventeen schools that participated in LDC in 2017—2018 did not continue 
with the program in 2018-2019. Therefore, the 2018-2019 sample includes 36 schools, 
compared to 53 schools in 2017-2018. 
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Table 6.2 


Summary Table Displaying Fidelity of Implementation by Metric and Year 


2017-2018 2018-2019 
% of % of schools 
schools that that met 
met fidelity Program fidelity Program 
threshold met threshold met 
Key component Indicator (n=53) _ fidelity? (n=36) _ fidelity? 
Key component Teacher attendance at weekly PLC 49 No 19 No 
1: Common meetings 
planning ome tot Amount of time spent on LDC during 15 No 22 No 
LDC professional are 
fi common planning time 

learning 
community with Exposure to LDC LEARN content 0 No 6 No 
synchronous during first instructional cycle 
coach support Exposure to LDC LEARN content 2 No 3 No 

during second instructional cycle 

Perceived effectiveness of 57 No 67 No 

engagement in PLC on teacher 

competencies 
Key component Coach comments on modules 57 Yes 67 Yes 
2: Asynenronous Coach formative peer review on 8 No 6 No 
support from 

modules 
LDC coaches 

Teacher perception of the helpfulness 57 Yes 64 Yes 

of coach written feedback on 

modules 
Key component Module editing 15 No 28 No 
ss JeaCnes . Module implementation 4 No 3 No 
implementation 
activities 
Key component Frequency of coach/teacher leader 34 No 56 No 
4: Leadership monthly meetings 
Ss avid at Administrator attendance at 32 No 19 No 
different levels ; Pa 

quarterly in-person administrator 

meetings 

Teacher leader attendance at 49 No 42 No 

quarterly in-person teacher leader 

meetings 

Principal mini-task observation 42 No 44 No 
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Overall, fidelity was consistent over the 2 years, with the program meeting fidelity on the 
same 2 of 14 indicators each year: coach comments on modules and teacher perception of the 
helpfulness of coach written feedback. Trends were mixed with an increase from 2017-2018 to 
2018-2019 in the proportion of schools meeting fidelity on 9 of the indicators, and a decrease 
or the same proportion on 5 of the indicators. Asynchronous support by coaches was a strength 
of implementation in 2017-2018 and it seemed to be even more so in 2018-2019. Other 
positive trends included an increase in the proportion of schools with teachers who reported 
engagement in the PLC was effective in improving their competencies and an increase in 
engagement around editing modules. Of concern was a decrease in the proportion of schools 
meeting attendance targets both for PLC sessions and administrator/teacher leader 
professional development sessions. 


6.7 Exploratory Analysis of the Relationship Between Implementation Metrics 
and Student Outcomes 


Here we explore the relationship between fidelity of implementation indicators and 
student ELA outcome scores at the school level. In order to examine these relationships, we 
produced school residuals, which are estimates of individual schools’ contributions to student 
ELA scores (i.e., schools’ effectiveness in impacting student achievement), holding the other 
variables in the analysis model constant. 


An MMMC model was used to produce individual estimates for each LDC school. The 
models utilized the analytic sample of Cohort 1 and Cohort 2 elementary and middle school 
teachers and students reported on in Chapter 7 (the largest sample for our primary quasi- 
experimental analyses) (see Table 7.3). That model measured outcomes for Cohort 1 teachers 
in 2017-2018 and Cohort 2 teachers in 2018-2019. The model we used to produce the school 
residuals was identical to our outcome models with two exceptions: (a) the LDC treatment 
variable was removed from the equation; and (b) the teacher classification level was removed. 


The school residuals were used as the estimates of effectiveness, after adjustment by all 
the model covariates. These residuals are sometimes referred to as best linear unbiased 
predictors (BLUPs) in the relevant literature (Robinson, 1991). Because the residuals are 
obtained from our analytic samples, which exclude many participating schools and teachers for 
a variety of reasons (e.g., lack of prior or outcome scores; teaching a subject other than ELA, 
social studies, or science), the resulting estimates represent a subset of all schools that 
participated in either 2017—2018 or 2018-2019. Residuals were produced for 29 out of 53 
schools (55%). 
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Table 6.3 


Mean School-Level Fidelity Scores by Indicator and Achievement Group 


Low- Middle- High- Difference 
achieving achieving achieving imieans 
schools schools schools significant 
(n = 10) (n=9) (n = 10) at 5% 
Key component Fidelity indicator M N M M N level 
Key component __ Teacher attendance at weekly 144 9 0.75 8 0.63 8 No 
1: Common PLC meetings 
planning time for amount of time spentonLDC 140 10 0.75 8 080 10 No 
te professional during common planning time 
earning 
community with isioals io LDC ee 0.40 10 0.00 9 0.10 10 No 
synchronous uring first instructional cycle 
coach support Exposure to LDC LEARN content 0.30 10 0.33 9 0.00 10 No 
during second instructional cycle 
Perceived effectiveness of 2.20 10 1.38 8 1.89 9 No 
engagement in PLC on teacher 
competencies 
Key component Coach comments on modules 2.00 10 1.50 8 2.00 10 No 
2: Asynchronous Coach formative peerreviewon 0.00 10 0.11 9 0.00 10 #No 
support from LDC produles 
coaches . 
Teacher perception of the 2.10 10 1.63 8 1.44 9 No 
helpfulness of coach written 
feedback on modules 
Key component Module editing 0.60 10 0.56 9 0.70 10 No 
a eGGHey, Module implementation 0.10 10 000 9 0.00 10 No 
implementation 
activities 
Key component Frequency of coach/teacher 2.00 10 0.75 8 1.70 10 Yes 
4: Leadership leader monthly meetings 
support at Administrator attendance at 0.70 10 100 9 1.10 10 #£4.No 
different levels quarterly in-person 
administrator meetings 
Teacher leader attendance at 1.80 10 1.22 9 1.60 10 No 
quarterly in-person teacher 
leader meetings 
Principal mini-task observation 2.10 10 0.63 8 1.00 9 Yes 


As with the outcome data, fidelity data for this exploratory analysis came from 2 years: 
2017-2018 for Cohort 1 schools and 2018-2019 for Cohort 2 schools. For the examination of 
the relationship between fidelity and outcomes, we included school-level versions of the 14 
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fidelity matrix indicators, each measured using a 4-point scale of 0 to 3. We examined the 
distribution of school-level residuals and divided them into three categories: ten low-achieving 
schools with residuals of less than -0.05, nine middle-achieving schools with residuals between - 
0.05 and 0.05, and 10 high-achieving schools with residuals greater than 0.05. We then 
produced mean implementation scores on each indicator for the three achievement groups, 
which are displayed in Table 6.3. A descriptive examination of the school means did not reveal 
any consistent pattern between the implementation scores and achievement levels. We also 
conducted a one-way ANOVA to test whether the differences in means between the three 
groups were significant. These tests only found statistically significant differences on two 
indicators, and these results may be an artifact of conducting multiple comparisons. In 
summary, the analysis provided no evidence of a link between school-level fidelity of 
implementation on the matrix indicators and school-level effectiveness. 


6.8 Summary of Results 


In summary, our analysis revealed that none of the schools met fidelity requirements on 
the four key components in 2018-2019 in New York. The ability of PLCs to set aside common 
planning time that worked for all teachers varied across schools. The frequency of meetings 
ranged broadly, as did the attendance rates of the participants. While a handful of schools were 
able to maintain high attendance rates, most did not meet the desired attendance threshold. 
When PLCs did meet, they most often met for 45 to 59 minutes, although many sessions also 
lasted the desired 60 minutes or more. LDC intended for teachers to be exposed to key online 
course content within PLC time, and this year’s fidelity analysis showed substantial 
improvement at the teacher level on these metrics. The vast majority of schools, however, still 
did not meet the adequate implementation thresholds for the instructional course exposure 
metrics. The proportion of schools that met fidelity in terms of teachers’ perception of the 
impact of LDC on their skills increased relative to the prior year; nevertheless, the program as a 
whole did not meet fidelity on this metric. 


Asynchronous support was a strength of implementation in New York. The program met 
fidelity in both years on coach comments on modules and teachers’ perception of the 
helpfulness of written feedback, and the proportion of schools meeting fidelity on these two 
indicators increased from 2017-2018 to 2018-2019. Peer review remained a little used 
function in CoreTools. 


Results on module editing suggest that many teachers were not heavily engaged in the 
design process, with half of teachers not having edited any teaching tasks. Those that did edit 
the teaching task tended to also edit either standards or texts, and a third of teachers also 
reached the stage of editing skills/mini-tasks and the rubric. There was evidence that about one 
quarter of teacher participants implemented at least one module in the classroom, with less 
than 20% of teachers meeting the adequate implementation threshold of uploading student 
work for two modules. There was some drop-off in engagement by both administrators and 
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teacher leaders in professional development meetings from 2017-2018 to 2018-2019. The 
program was somewhat more successful in 2018-2019 in ensuring regular contact between 
teacher leaders and coaches and broad engagement by administrators in observing LDC 
classroom implementation, but overall fidelity was still not met on these indicators. 


Variation across schools was substantial. Some schools met fidelity on seven to eight 
indicators, and on the other end, some schools struggled with nearly every aspect of 
implementation. No strong patterns emerged for the cohort and school-level subgroups with 
Cohort 1 and Cohort 2 schools, as well as the elementary and secondary schools meeting similar 
levels of fidelity, despite great variation within each of the subgroups. 


Despite substantial variation across schools and teachers in fidelity of implementation 
across the different metrics, an exploratory analysis did not find any relationship at the school 
level between fidelity and effectiveness in impacting student ELA scores. 
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7.0 Student Outcome Analysis 


This section presents the student outcome analysis we conducted to evaluate LDC’s 
impact on student learning in the 2016-2017, 2017-2018, and 2018-2019 school years. As 
described earlier, our LDC teacher sample included both elementary and middle school 
teachers from two cohorts of LDC schools in NYCDOE. Our primary analyses examine the impact 
of LDC as practiced by teachers participating in the program in two consecutive years. These 
analyses pooled teachers and students from both cohorts of schools, and were conducted for 
all elementary school teachers, all middle school teachers, and all teachers combined across 
school levels. Each separate analysis included matching and regression stages. 


Another primary analysis examined the impact of Cohort 1 teachers in their third 
consecutive year of LDC implementation. Additionally, when there was sufficient sample, we 
conducted supplemental analyses examining cohorts separately and examining teachers who 
joined existing LDC PLCs in later years. 


The following list describes the analyses conducted, including details on cohort, school 
level, the number of consecutive years that teachers implemented LDC, and the outcome year: 


1. Cohorts 1 and 2 elementary school sample: Teachers implementing LDC for 2 
consecutive years (Cohort 1: 2016-2017 and 2017-2018; Cohort 2: 2017-2018 and 
2018-2019) 


2. Cohorts 1 and 2 middle school sample: Teachers implementing LDC for 2 consecutive 
years (Cohort 1: 2016-2017 and 2017-2018; Cohort 2: 2017-2018 and 2018-2019) 


3. Cohorts 1 and 2 elementary and middle school sample: Teachers implementing LDC for 
2 consecutive years (Cohort 1: 2016-2017 and 2017-2018; Cohort 2: 2017-2018 and 
2018-2019) 


4. Cohort 1 middle school sample: Teachers implementing LDC for 3 consecutive years 
(2016-2017, 2017-2018, and 2018-2019) 


5. Cohort 1 elementary school sample: Teachers implementing LDC for 1 year (2016-— 
2017) 


6. Cohort 1 middle school sample: Teachers implementing LDC for 1 year (2016-2017) 


7. Cohort 1 middle school sample: Teachers implementing LDC for 2 consecutive years 
(2017-2018 and 2018-2019) 


8. Cohort 1 middle school sample: Teachers implementing LDC for 1 year (2017-2018) 


9. Cohort 2 elementary school sample: Teachers implementing LDC for 1 year (2017- 
2018) 


10. Cohort 2 elementary school sample: Teachers implementing LDC for 2 consecutive 
years (2017-2018 and 2018-2019) 
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11. Cohort 2 middle school sample: Teachers implementing LDC for 1 year (2017-2018) 


12. Cohort 2 middle school sample: Teachers implementing LDC for 2 consecutive years 
(2017-2018 and 2018-2019) 


13. Cohort 2 middle school sample: Teachers implementing LDC for 1 year (2018-2019) 


14. Additional analyses based on the Cohorts 1 and 2 middle school sample: LDC students 
with or without prior exposure to LDC teachers and LDC students with high or low LDC 
dosage in the outcome year 


LDC’s theory of action predicts that it will take 2 years of experience with LDC before 
teachers have a measurable impact on student learning. As such, the first four analyses listed 
constitute this study’s primary tests of the impact of LDC. Analyses numbered 5 through 14 are 
considered supplementary. All supplementary analyses, with the exception of the 14°", examine 
Cohorts 1 and 2, and elementary and middle school separately. Analyses explore impacts after 
1 and 2 years of teacher participation, and explore the effectiveness of teachers who joined LDC 
as it was being launched at their schools and later in the initiative. The 14th set of analyses 
break the Cohorts 1 and 2 middle school sample (listed as number 2 above) into subgroups 
based on whether students were exposed to LDC in the prior year and how much LDC 
instruction they were exposed to in the outcome year. The complete set of sample, matching, 
and analysis results tables for each supplemental analysis are presented in Appendix I. 


We could not conduct analyses for certain groups of teachers because of sample sizes. 
More specifically, we could not conduct meaningful analyses of the impact of Cohort 1 
elementary teachers who joined in 2016-2017 in their second year of implementation (2017— 
2018) or Cohort 1 elementary teachers who joined LDC in 2017-2018 or 2018-2019. We were 
also unable to conduct meaningful analyses of Cohort 1 middle school teachers or Cohort 2 
elementary teachers who joined LDC in 2018-2019. 


We begin this chapter by describing the process we used to define the LDC student 
samples for each analysis and to construct matched comparison samples. We then present 
descriptive statistics for the treatment and comparison groups. Next, we report the estimated 
impact of LDC on students as measured by their New York State Assessment ELA scores for our 
primary and supplementary analyses. Finally, we summarize and contextualize the results. 


7.1 LDC Sample and the Matching Process 


For each analysis, the eligible LDC sample includes all students (a) who were enrolled in 
one school campus for the entire outcome year under the instruction of at least one of the 
participating LDC teachers and (b) for whom baseline achievement scores, outcome year 
achievement scores, and demographic data were available. Achievement and demographic 
data were used in the matching process. Matching was conducted for each sample/analysis. 
Here we include the results of the matching for the primary analysis samples (elementary 
teachers with 2 years of LDC, middle school teachers with 2 years of LDC, combined elementary 
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and middle school teachers with 2 years of LDC, and Cohort 1 middle school teachers with 3 
years of LDC). The matching results for the supplementary analyses are included in Appendix I. 


Cohorts 1 and 2 Elementary School Sample (Teachers with 2 years of Implementation) 


Across Cohorts 1 and 2, there were 11 LDC schools serving elementary grades and 19 LDC 
teachers with 2 years of LDC implementation in their classrooms. These 19 teachers were linked 
to 388 eligible students prior to the CEM process. After the student-level matching, our final 
pooled elementary LDC sample included 363 students and the same number of teachers and 
schools prior to matching (see Table 7.1). 


Prior to matching, the potential elementary school comparison sample consisted of 781 
schools, 8,149 teachers, and 112,663 students. This comparison sample was substantially 
reduced during the first stage of matching, which identified up to five schools that closely 
resembled each of the LDC schools. After student-level CEM, a workable analytic comparison 
sample consisted of 47 schools, 274 teachers, and 363 students. 


Table 7.1 


Before and After Matching Sample Sizes: Cohorts 1 and 2 Elementary School Analysis for Teachers with 2 
Years of LDC Implementation 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 11 19 388 781 8,149 112,663 
After school matching 11 19 388 54 432 3,662 
Stage 2 
After student matching 11 19 363 47 274 363 


Cohorts 1 and 2 Middle School Sample (Teachers with 2 Years of Implementation) 


As reported in Table 7.2, the resulting LDC sample for the pooled middle school teachers 
with 2 years of implementation included 19 schools, 48 teachers, and 2,508 students prior to 
the CEM process. After the CEM student-level matching, our final pooled middle school LDC 
sample was reduced to 2,355 students. 


Prior to matching, the potential comparison sample consisted of 529 schools, 12,194 
teachers, and 213,393 students. This comparison sample was substantially reduced during the 
first stage of matching, which identified up to five schools that most closely resembled each of 
the LDC schools. After student-level CEM, a workable analytic comparison sample of 74 schools, 
1,212 teachers, and 2,355 students was constructed. 
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Table 7.2 


Before and After Matching Sample Sizes: Cohorts 1 and 2 Middle School Analysis for Teachers with 
2 years of LDC Implementation 


LDC sample Comparison sample 
Stage School Teacher Student School Teacher Student 
Stage 1 
Before school matching 19 48 2,508 529 12,194 213,393 
After school matching 19 48 2,508 79 1,929 24,076 
Stage 2 
After student matching 19 48 2,355 74 1,212 2,355 


Cohorts 1 and 2 Elementary and Middle School Sample (Teachers with 2 Years of 
Implementation) 


We also conducted an analysis which pooled teachers with 2 years of LDC experience 
across cohorts and school levels. As shown in Table 7.3, the resulting LDC sample included 29 
schools, 66 teachers, and 2,896 students prior to the CEM process. Note that one LDC school 
with teachers who participated in two consecutive years spanned elementary and secondary 
grades. After the CEM student-level matching, our final secondary LDC sample was reduced to 
2,718 students. 


Table 7.3 


Before and After Matching Sample Sizes: Cohorts 1 & 2 Elementary and Middle School Analysis for 
Teachers with 2 years of LDC Implementation 


LDC sample Comparison sample 
Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 29 66 2,896 1,121 19,597 281,643 
After school matching 29 66 2,896 129 2,216 27,651 
Stage 2 
After student matching 29 66 2,718 118 1,475 2,718 


Prior to matching, the potential comparison sample consisted of 1,121 schools, 19,597 
teachers and 281,643 students. This comparison sample was substantially reduced during the 
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first stage of matching, which identified up to five schools that closely resembled each of the 
LDC schools. After student-level CEM, a workable analytic comparison sample of 118 schools, 
1,475 teachers, and 2,718 students was constructed. 


Cohort 1 Middle School Sample for Teachers with 3 Years of LDC Implementation 


For Cohort 1 middle schools that started their LDC implementation in 2016-2017, we 
conducted an additional analysis examining the impact of teachers who had 3 years of LDC 
experience. As shown in Table 7.4, the resulting Cohort 1 LDC middle school sample included six 
schools, eleven teachers, and 451 students prior to the CEM process. After the CEM student- 
level matching, our final secondary LDC sample was reduced to 406 students. 


Table 7.4 


Before and After Matching Sample Sizes: Cohort 1 Middle School Analysis for Teachers with 3 Years of 
LDC Implementation 


LDC sample Comparison sample 
Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 6 11 451 480 9,965 153,981 
After school matching 6 11 451 25 471 5,613 
Stage 2 
After student matching 6 11 406 23 271 406 


Prior to matching, the potential comparison sample consisted of 480 schools, 9,965 
teachers and 153,981 students. This comparison sample was substantially reduced during the 
first stage of matching, which identified up to five schools that closely resembled each of the 
LDC schools. After student-level CEM, a workable analytic comparison sample of 23 schools, 
271 teachers, and 406 students was constructed. 


7.2 Primary Outcome Analysis: Descriptive Results on the Matched Analytic 
Samples 


Tables 7.5 through 7.8 present the characteristics of the LDC students and comparison 
students in the four sets of primary analyses respectively. Treatment and comparison samples 
matched very closely. Exact matching was achieved on some variables, and all demographic 
variables were within five percentage points. We used the spring 2019 New York State 
Assessment ELA scores as our outcome measures in all analyses except when we combined 
Cohorts 1 and 2 schools; for those pooled analyses, spring 2018 test scores were used as the 
outcome measures for Cohort 1 students. Additionally, the Cohort 1 analyses used the spring 
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2016 New York State Assessment ELA scores as one of the matching variables while the Cohort 
2 analyses used the spring 2017 New York State Assessment ELA scores. 


For the baseline and outcome achievement variable, we standardized New York State 
Assessment ELA scale scores at each grade level relative to district performance, based on the 
district mean and standard deviation for the ELA test at each grade level. Standardizing scores 
in this way enabled us to easily compare our samples’ performance relative to the district’s and 
to compare scores across grades and years more easily. A standardized scale score of zero, for 
example, indicates that the student scored at the mean for all other students in the district who 
took the same test. A standardized scale score of 1.0 meant that the student scored one 
standard deviation higher than the district mean. Conversely, a standardized scale score of -1.0 
indicated that the student scored one standard deviation lower than the district mean. 


The pooled LDC elementary school student sample after matching was composed mainly 
of Hispanic and Black students who were economically disadvantaged (see Table 7.5). English 
language learners represented 15.4% of this sample, while special education students 
represented another 15.4%. The sample was also comprised of students in third grade (100.0%) 
in the baseline year (and therefore in fifth grade in the outcome year). Mean performance on 
the baseline year assessments was about 30% of a standard deviation below the districtwide 
performance levels in both mathematics and ELA. 


The pooled LDC middle school student sample (see Table 7.6) was mostly composed of 
Hispanic and Black students, and 83.6% of the middle school students were economically 
disadvantaged. English language learners represented one eighth of this sample, while special 
education students represented about 25% of the sample. In addition, mean performance on 
the baseline year assessments was slightly lower for LDC students, as compared to districtwide 
performance levels in mathematics and ELA. 


Table 7.7 reports the sample information for the students taught by the final pooled LDC 
sample across both cohorts and school levels. As can be seen, this sample included a very large 
proportion of Black and Hispanic students (86.6%), and students who were economically 
disadvantaged (84.4%). English language learners represented about one eighth of this sample, 
while special education students were roughly a quarter of the sample. A majority of the 
sample was composed of students in middle school grades during the outcome year. In 
addition, mean performance on the baseline year assessment was slightly lower for LDC 
students as compared to districtwide performance levels in mathematics and ELA. 


Table 7.8 reports the sample information for the students taught by Cohort 1 middle 
school teachers with 3 years of LDC. This LDC sample consisted mostly of Black and Hispanic 
students (93.1%) and students who were economically disadvantaged (87.4%). English language 
learners represented slightly more than one sixth of this sample, and about one fifth of the 
students were in special education. In addition, mean performance on the baseline year 
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assessments was about one half of a standard deviation lower for LDC students as compared to 
districtwide performance levels in mathematics and ELA. 


Table 7.5 


Baseline Characteristics of the Pooled Group of Treated Students Taught by Cohort 1 and 2 Elementary 
School Teachers Participating in LDC in Two Consecutive Years and Comparison Students After Matching 


Student characteristic 


Treatment group 


Comparison group 


(n = 363) (n = 363) 

Race/ethnicity 

Hispanic (%) 60.3 60.3 

Black (%) 34.4 34.7 

Asian (%) 4.1 4.1 

White (%) 0.3 0.6 

Other (%) 0.8 0.3 
Female (%) 47.7 47.7 
Special programs status 

Poverty (%) 90.1 90.4 

English language learner (%) 15.4 16.3 

Special education (%) 15.4 15.4 
Student baseline achievement 

Mean baseline year mathematics Z score -0.290 -0.313 

Mean baseline year ELA Z score -0.262 -0.276 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.286 -0.300 

Teacher years of experience 13.5 13.4 
Grade level at baseline year 

Grade 3 (%) 100 100 
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Table 7.6 


Baseline Characteristics of the Pooled Group of Treated Students Taught by Cohort 1 and 2 Middle School 
Teachers Participating in LDC in Two Consecutive Years and Comparison Students After Matching 


Student characteristic 


Treatment group 


Comparison group 


(n = 2,355) (n = 2,355) 

Race/ethnicity 

Hispanic (%) 45.6 45.6 

Black (%) 39.7 39.7 

Asian (%) 6.2 5.4 

White (%) 7.3 8.1 

Other (%) 1.2 1.2 
Female (%) 50.9 51.4 
Special programs status 

Poverty (%) 83.6 83.7 

English language learner (%) 12.7 13.2 

Special education (%) 25.3 23.9 
Student baseline achievement 

Mean baseline year mathematics Z score -0.458 -0.434 

Mean baseline year ELA Z score -0.374 -0.376 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.404 -0.364 

Teacher years of experience 8.8 8.4 
Grade level at baseline year 

Grade 4 (%) 25.4 25.4 

Grade 5 (%) 40.7 40.7 

Grade 6 (%) 33.9 33.9 
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Table 7.7 


Baseline Characteristics of the Pooled Group of Treated Students Taught by Cohort 1 and 2 Elementary 
and Middle School Teachers Participating in LDC in Two Consecutive Years and Comparison Students 


After Matching 


Student characteristic 


Treatment group 


Comparison group 


(n = 2,718) (n = 2,718) 

Race/ethnicity 

Hispanic (%) 47.6 47.6 

Black (%) 39.0 39.1 

Asian (%) 5.9 5.2 

White (%) 6.3 Fi. 

Other (%) 1.2 1.1 
Female (%) 50.5 50.9 
Special programs status 

Poverty (%) 84.4 84.6 

English language learner (%) 13.1 13.6 

Special education (%) 24.0 22.7 
Student baseline achievement 

Mean baseline year mathematics Z score -0.436 -0.418 

Mean baseline year ELA Z score -0.359 -0.363 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.388 -0.356 

Teacher years of experience 9.4 9.1 
Grade level at baseline year 

Grade 3 (%) 13.4 13.4 

Grade 4 (%) 22.0 22.0 

Grade 5 (%) 35.3 35.3 

Grade 6 (%) 29.4 29.4 
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Table 7.8 


Baseline Characteristics of the Pooled Group of Treated Students Taught by Cohort 1 Middle School 
Teachers Participating in LDC in Three Consecutive Years and Comparison Students After Matching 


Student characteristic area a 

Race/ethnicity 

Hispanic (%) 44.3 44.3 

Black (%) 48.8 52.5 

Asian (%) 6.2 2.2 

White (%) 0.3 0.3 

Other (%) 0.4 0.7 
Female (%) 72.2 72.2 
Special programs status 

Poverty (%) 87.4 85.5 

English language learner (%) 17.5 13.3 

Special education (%) 19.2 19.2 
Student baseline achievement 

Mean baseline year mathematics Z score -0.557 -0.555 

Mean baseline year ELA Z score -0.406 -0.416 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.417 -0.418 

Teacher years of experience 8.0 7.9 
Grade level at baseline year 

Grade 3 (%) 19.5 19.5 

Grade 4 (%) 32.0 32.0 

Grade 5 (%) 48.5 48.5 


7.3 Primary Outcome Analysis Results: Teachers Participating in LDC for Two 
Consecutive Years Across Cohorts 


Here, we examine the impact of LDC as practiced by teachers with 2 consecutive years of 
experience with the program. We first present the results for the elementary and middle 
schools separately, after which we present the results of an analysis pooling the two groups of 
schools together. As discussed in detail in Chapter 2, students could be exposed to between 
zero and three different LDC teachers at the elementary school level. In contrast, middle school 
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students could be exposed to between zero and six different LDC teachers in a given year. Using 
a MMMC design, each observation at Level 1 represented one student. Weights across teachers 
for each student summed to a unity (1). 


Two different approaches were used to model the LDC treatment intervention variable as 
a fixed effect at the student level. The first model was dosage dependent, and considered 
variation in students’ level of exposure to LDC teachers. In this approach, the treatment 
variable was structured as a continuous response variable between zero and one, based on 
exposure to LDC teachers. In contrast, the second approach was modeled as dosage 
independent, and considered any student exposed to any LDC intervention teacher to be a 
treated individual. In this latter approach the treatment variable was dichotomous, coded as 
one for LDC treated students and zero for comparison students. 


Table 7.9 


Effect Estimates of Cohort 1 and 2 Elementary School Teachers Participating in LDC in Two Consecutive 
Years on New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


aeapiee Dosage-dependent model Dosage-independent model 
coefficient (SD) coefficient (SD) 
Level 2 LDC teacher treatment 0.041 (0.139) 0.025 (0.091) 


Level 1 student characteristics 


Hispanic 

Black 

Other ethnicity 

Poverty 

Female 

English language learner 
Special education 
Teacher experience 


Baseline peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


Cohort 2 schools 


-0.099 (0.126) 
-0.258 (0.130)* 
-0.372 (0.357) 
-0.049 (0.086) 
0.134 (0.053)* 
0.058 (0.079) 
-0.336 (0.077)* 
0.011 (0.007) 
0.073 (0.081) 
0.198 (0.043)* 
0.560 (0.046)* 
0.216 (0.140) 


-0.100 (0.125) 
-0.259 (0.130)* 
-0.365 (0.357) 
-0.049 (0.086) 
0.134 (0.053)* 
0.058 (0.079) 
-0.336 (0.077)* 
0.011 (0.007) 
0.074 (0.081) 
0.198 (0.043)* 
0.560 (0.046)* 
0.219 (0.140) 


Note. Based on the dosage-dependent model, the average treated student received a 0.567 treatment 


dosage. Because of this, using the dosage-dependent model we could estimate the ATET at (0.567 


x 0.041) = 0.023. 
*Cl two tailed probability > .95 


In Table 7.9, we present results of both the dosage dependent and dosage independent 
models, estimating the impact of LDC on Cohort 1 and Cohort 2 elementary students (Grade 5 
students in the outcome year). As can be seen, model results for the LDC effect on elementary 
student outcomes are in the positive direction, but are not statistically significant for either 
model. In other words, neither model provided sufficient evidence to conclude that elementary 
students taught by LDC teachers performed better on the ELA assessment than did their 
matched peers in the comparison group. 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Baseline ELA performance was the strongest 
predictor and baseline mathematics performance also helped explain the outcome. In addition 
to baseline achievement, three student level demographic variables helped predict 
performance statistically and significantly: 1) Black students performed at significantly lower 
levels than students who were White or Asian, 2) females performed at higher levels than did 
males, and 3) special education students performed at lower levels than regular students. 


In Table 7.10, we present results of both the dosage dependent and dosage independent 
models, estimating the impact of LDC on Cohort 1 and Cohort 2 students in middle schools. As 
can be seen, model results for the LDC effect on middle school student outcomes are in the 
positive direction, but are not statistically significant for either model. As with the elementary 
analysis, neither model provided sufficient evidence to conclude that middle school students 
taught by LDC teachers performed better on the ELA assessment than did their matched peers 
in the comparison group. 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Baseline ELA performance was the strongest 
predictor and baseline mathematics performance also helped explain the outcome. In addition 
to baseline achievement, four student level demographic variables helped predict performance 
statistically and significantly: 1) Black students had lower scores than White students, 

2) students who were economically disadvantaged had lower scores than did students who 
were not, 3) females had higher scores than males, and 4) special education students had lower 
scores than students who were not classified as special education. 
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Table 7.10 


Effect Estimates of Cohort 1 and 2 Middle School Teachers Participating in LDC in Two Consecutive Years 
on New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables 


Dosage-dependent model 
coefficient (SD) 


coefficient (SD) 


Dosage-independent model 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 

Black 

Asian 

Other ethnicity 

Poverty 

Female 

English language learner 
Special education 

Teacher experience 


Baseline peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 
Grade 5 at baseline 
Grade 6 at baseline 


Cohort 2 schools 


0.005 (0.092) 


-0.048 (0.042) 
-0.135 (0.044)* 
0.081 (0.054) 
0.141 (0.092) 
-0.093 (0.026)* 
0.178 (0.019)* 
-0.021 (0.032) 
-0.107 (0.025)* 
0.002 (0.003) 
0.296 (0.026)* 
0.178 (0.015)* 
0.564 (0.016)* 
0.047 (0.031) 
0.095 (0.033)* 
0.026 (0.035) 


0.020 (0.038) 


-0.048 (0.042) 
-0.135 (0.044)* 
0.081 (0.054) 
0.142 (0.092) 
-0.093 (0.026)* 
0.178 (0.019)* 
-0.021 (0.032) 
-0.107 (0.025)* 
0.002 (0.003) 
0.297 (0.026)* 
0.179 (0.015)* 
0.564 (0.016)* 
0.048 (0.031) 
0.095 (0.033)* 
0.026 (0.035) 


Note. Based on the dosage-dependent model, the average treated student received a 0.341 treatment 


dosage. Because of this, using the dosage-dependent model we could estimate the ATET at (0.341 


x 0.005) = 0.002. 
*Cl two tailed probability > .95 


Table 7.11 displays the dosage dependent and dosage independent analysis results when 


we combine elementary and middle school students from both cohorts. As can be seen, model 
results for the LDC effect on elementary and middle school student outcomes are in the 
positive direction, but are not statistically significant for either model. In other words, neither 
model provided sufficient evidence to conclude that elementary and middle school students 
taught by LDC teachers performed better on the ELA assessment than did their matched peers 
in the comparison group. 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Baseline ELA performance remained the 
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strongest predictor and baseline mathematics performance also helped explain the outcome. In 


addition to baseline achievement, Black, female, poverty, and special education were all 
significant predictors of ELA performance and were in the expected directions. 


Table 7.11 


Effect Estimates of Cohort 1 and 2 Elementary and Middle School Teachers Participating in LDC in Two 


Consecutive Years on New York State ELA Assessment, Dosage-Dependent and Dosage-Independent 


Models 


Variables 


Dosage-dependent model 


coefficient (SD) 


Dosage-independent model 


coefficient (SD) 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 

Black 

Poverty 

Female 

English language learner 
Special education 

Teacher Experience 

Baseline peer ELA Z score 
Baseline year mathematics Z score 
Baseline year ELA Z score 
Grade 4 baseline 

Grade 5 baseline 

Grade 6 baseline 


Cohort 2 schools 


0.019 (0.075) 


-0.047 (0.042) 
-0.145 (0.044)* 
-0.091 (0.025)* 
0.174 (0.018)* 
-0.013 (0.030) 
-0.128 (0.024)* 
0.004 (0.003) 
0.276 (0.025)* 
0.179 (0.014)* 
0.563 (0.015)* 
-0.071 (0.047) 
-0.024 (0.045) 
0.022 (0.046) 
0.029 (0.034) 


0.020 (0.035) 


-0.046 (0.042) 
-0.145 (0.044)* 
-0.091 (0.025)* 
0.174 (0.018)* 
-0.014 (0.030) 
-0.128 (0.024)* 
0.004 (0.003) 
0.276 (0.025)* 
0.179 (0.014)* 
0.563 (0.015)* 
-0.072 (0.048) 
-0.024 (0.045) 
0.022 (0.046) 
0.028 (0.034) 


Note. Based on the dosage-dependent model, the average treated student received a 0.371 treatment 
dosage. Because of this, using the dosage-dependent model we could estimate the ATET at 

(0.371 x 0.019) = 0.007. 

*Cl two tailed probability > .95 


7.4 Primary Outcome Analysis Results: Cohort 1 Teachers Participating in LDC 
for Three Consecutive Years 


In this section, we examine the impact of LDC as practiced by middle school teachers with 
3 consecutive years of experience with the program (the elementary analysis is not possible 
because of the 3-year gap between the baseline and outcome year, and because outcome 
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scores are only available for third grade and higher). As can be seen in Table 7.12, model results 
for the LDC effect on student outcomes were in the positive direction, but were not statistically 
significant for either model. In other words, neither model provided sufficient evidence to 
conclude that middle school students taught by LDC teachers with 3 years of LDC experience 
performed better on the ELA assessment than did their matched peers in the comparison 


group. 


Table 7.12 


Effect Estimates of Cohort 1 Middle School Teachers Participating in LDC in Three Consecutive Years on 
New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Wariables Dosage-dependent model Dosage-independent model 
coefficient (SD) coefficient (SD) 
Level 2 LDC teacher treatment 0.125 (0.250) 0.010 (0.090) 


Level 1 student characteristics 


Hispanic 

Black 

Poverty 

Female 

English language learner 
Special education 
Teacher experience 


Baseline peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 
Grade 4 at baseline 


Grade 5 at baseline 


-0.318 (0.118)* 
-0.472 (0.113)* 
-0.077 (0.072) 
0.190 (0.061)* 
0.154 (0.075)* 
-0.159 (0.069)* 
-0.006(0.008) 
0.394 (0.068)* 
0.141 (0.038)* 
0.507 (0.041)* 
0.035 (0.079) 
0.092 (0.076) 


-0.318 (0.118)* 
-0.471 (0.114)* 
-0.077 (0.072) 
0.191 (0.061)* 
0.154 (0.075)* 
-0.158 (0.069)* 
-0.005(0.008) 
0.396 (0.068)* 
0.142 (0.038)* 
0.507 (0.041)* 
0.034 (0.079) 
0.091 (0.076) 


Note. Based on the dosage-dependent model, the average treated student received a 0.268 treatment 


dosage. Because of this, using the dosage-dependent model we could estimate an ATET at (0.268 


x 0.125) = 0.034. 
*Cl two tailed probability > .95 


The significant effects of the covariates on student performance were similar under both 
models and were in the expected direction. Once again, baseline ELA performance was the 
strongest predictor and baseline mathematics performance also helped explain the outcome. In 
addition, Black and Hispanic students performed at lower levels than did their peers, female 
students performed higher, and special education students performed at a lower level than 
students not classified as special education. One interesting result was that English language 
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learners in this analysis were found to have statistically significant higher ELA scores than non- 


English language learner students. 


7.5 Supplementary Outcome Analysis Results: Cohort 1 Elementary Teachers 


Here we report the effect of LDC as practiced by Cohort 1 elementary teachers in their 


first year of participation, 2016-2017. Due to small sample sizes, no analyses were conducted 


for the Cohort 1 elementary teachers who newly joined the LDC program in 2017-2018, for 


those Cohort 1 elementary teachers who implemented LDC in both 2016-2017 and 2017-2018, 


or for Cohort 1 elementary teachers who joined LDC in 2018-2019. 


Table 7.13 


Effect Estimates of Cohort 1 Elementary School Teachers Participating in LDC for One Year on 2016— 


2017 New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables 


Dosage Dependent 
model coefficient (SD) 


Dosage independent 
model coefficient (SD) 


Level 1 - LDC teacher treatment 


Level 2 - Student characteristics 


Hispanic 

Black 

Asian 

Other 

Poverty 

Female 

English language learner 
Special Education 

Grade 4 at baseline 
Teacher experience 


Baseline peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


0.084 (0.147) 


-0.110(0.163) 
-0.195 (0.152) 
-0.143 (0.148) 
-0.306 (0.208) 
-0.058 (0.065) 
0.171 (0.053)* 
-0.034 (0.107) 
-0.169 (0.089) 
-0.121 (0.105) 
0.001 (0.010) 
0.152 (0.092) 
0.166 (0.044)* 
0.528 (0.051)* 


-0.059 (0.118) 


-0.113(0.163) 
-0.191 (0.152) 
-0.144 (0.148) 
-0.302 (0.208) 
-0.055 (0.065) 
0.169 (0.053)* 
-0.030 (0.107) 
-0.168 (0.089) 
-0.126 (0.104) 
0.001 (0.010) 
0.144 (0.092) 
0.167 (0.044)* 
0.528 (0.051)* 


Note. Since the average treatment student received a 0.724 treatment dosage we could estimate an 


average treatment effect on the treated (ATET) at 0.724 * 0.084 = 0.061. This effect was not statistically 


significant. Inclusion rules led to removal of ELL students and students with disabilities variables from 


the model. 
*Cl two tailed probability > .95 
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Table 7.13 reports the model results for the effect of LDC teachers in 2016-2017 on 
student outcomes (matching sample sizes and results can be found in Tables I1 and 12). The 
outcome variable for both models was students’ 2017 New York State assessment scores in 
ELA. As reported, the dosage dependent effect is in the positive direction and the dosage 
independent effect is in the negative direction, but neither effect was statistically significant. In 
other words, neither model provided sufficient evidence to conclude that students taught by 
LDC teachers performed at levels different than their matched peers in the comparison group. 


The effects of the covariates on student performance were similar under both models and 
were in the expected directions (although most were not statistically significant). Prior ELA 
performance was the strongest predictor and prior mathematics performance also helped 
explain the outcome. In addition to prior achievement, one demographic variable (gender) 
helped predict performance. Females performed at significantly higher levels than males. 


7.6 Supplementary Outcome Analysis Results: Cohort 1 Middle School Teachers 


For Cohort 1 middle schools, we conducted analysis for teachers who implemented LDC in 
2016-2017, who joined existing PLCs in 2017-2018, and who implemented in both 2016-2017 
and 2017-2018. Due to small sample size, no analysis was conducted for teachers who newly 
joined LDC PLCs in Cohort 1 middle schools in 2018-2019. 


In Table 7.14 we present results of both the dosage dependent and dosage independent 
models on middle school students’ ELA performance in 2016-2017 after being taught by 
teachers in their first year of LDC implementation (matching sample sizes and results can be 
found in Tables 14 and |5). Model results show no statistically discernible LDC effect on student 
outcomes for either model. In other words, both analyses showed that students taught by LDC 
teachers scored similarly on the ELA test to their matched peers in the comparison group. The 
effects of the covariates on student performance were similar in direction to those in the 
elementary school models for the same cohort and outcome year, although due to the larger 
sample size more demographic variables were significant in the middle school analyses. Prior 
ELA and mathematics performance, as well as ELL status, special education status, Black 
ethnicity, Other Ethnicity, and gender were significant predictors of ELA performance. 
Additionally, the average prior achievement of a student’s peers was also a significant predictor 
of ELA performance in the expected direction. 
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Table 7.14 


Effect Estimates of Cohort 1 LDC Middle School Teachers Participating in LDC for One Year on 2016-2017 
New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables 


Dosage-dependent model 


coefficient (SD) 


coefficient (SD) 


Dosage-independent model 


Level 2 - LDC teacher treatment 


Level 1 - student characteristics 


Hispanic 

Black 

Asian 

Other 

Poverty 

Female 

English language learner 
Special education 

Grade 6 at baseline 
Grade 7 at baseline 
Content Exposure Weeks 
Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


-0.016 (0.066) 


-0.069 (0.036) 
-0.126 (0.038)* 
0.047 (0.046) 
-0.165 (0.083)* 
-0.039 (0.022) 
0.094 (0.015)* 
-0.101 (0.027)* 
-0.147 (0.020)* 
0.054 (0.032) 
0.108 (0.035)* 
-0.003 (0.002) 
-0.005 (0.003) 
0.155 (0.030)* 
0.179 (0.012)* 
0.658 (0.013)* 


-0.019 (0.042) 


-0.069 (0.036) 
-0.126 (0.038)* 
0.047 (0.046) 
-0.165 (0.083)* 
-0.039 (0.022) 
0.094 (0.015)* 
-0.101 (0.027)* 
-0.147 (0.020)* 
0.054 (0.032) 
0.108 (0.035)* 
-0.003 (0.002) 
-0.005 (0.003) 
0.155 (0.030)* 
0.179 (0.012)* 
0.658 (0.013)* 


Note. Since the average treatment student received a 0.506 treatment dosage we could estimate an 
average treatment effect on the treated (ATET) at 0.506 * -0.016 = -0.008. This effect was not 
statistically significant. 

*Cl two tailed probability > .95 


We also conducted student outcome analysis based on the 2017-2018 ELA scores for 
those students taught by LDC teachers who had 2 years of experience with LDC (matching 
sample sizes and results can be found in Tables I7 and 18). As can be seen in Table 7.15, model 
results for the LDC effect on student outcomes are in the positive direction, but are not 
Statistically significant for either model. In other words, neither analysis provided sufficient 
evidence to conclude that students taught by LDC teachers performed better on the ELA test 
than did their matched peers in the comparison group. 
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Table 7.15 


Effect Estimates of Cohort 1 Middle School Teachers Participating in LDC in Two Consecutive Years on 
2017-2018 New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables Dosage-dependent model Dosage-independent model 


coefficient (SD) 


coefficient (SD) 


Level 2 - LDC teacher treatment 
Level 1 - student characteristics 

Hispanic 

Black 

Asian 

Other 

Poverty 

Female 

English language learner 

Special education 

Grade 5 at baseline 

Grade 6 at baseline 

Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


0.001 (0.133) 


-0.084 (0.049) 
-0.180 (0.053)* 
0.110 (0.073) 
0.147 (0.123) 
-0.114 (0.035)* 
0.195 (0.025)* 
-0.008 (0.040) 
-0.104 (0.032)* 
0.107 (0.042)* 
0.112 (0.045)* 
0.002 (0.004) 
0.284 (0.035)* 
0.179 (0.019)* 
0.579 (0.021)* 


-0.009 (0.056) 


-0.084 (0.049) 
-0.180 (0.054)* 
0.109 (0.073) 
0.147 (0.123) 
-0.114 (0.035)* 
0.195 (0.025)* 
-0.008 (0.040) 
-0.104 (0.032)* 
0.108 (0.042)* 
0.112 (0.045)* 
0.002 (0.004) 
0.283 (0.035)* 
0.179 (0.019)* 
0.579 (0.021)* 


Note. Based on the dosage-dependent model, the average treated student received a 0.344 treatment 
dosage. Because of this, using the dosage dependent model we estimate an average treatment effect on 
the treated (ATET) at (0.345 * 0.001) = 0.0003. 

*Cl two tailed probability > .95 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Prior ELA performance was the strongest 
predictor and prior mathematics performance also helped explain the outcome. In addition to 
prior achievement, a number of demographic variables helped predict performance: 1) Black 
students performed at lower levels than White students, 2) students in special education 
performed at lower levels than students not in special education, 3) females performed at 
significantly higher levels than males, and 4) students with low socio-economic status 
performed at lower levels than did their peers with higher socio-economic status. 


In Table 7.16 we present the analysis results for students taught by Cohort 1 teachers who 
newly joined existing LDC PLCs in 2017-2018 (these teachers had 1 year of LDC experience but 
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their schools were in their second year of LDC implementation). Matching sample sizes and 
results can be found in Tables 110 and I11. As can be seen in Table 7.16, the LDC effect on 
student ELA outcomes are in the negative direction, but are not statistically significant for 
either model. In other words, neither model provided sufficient evidence to conclude that 
students taught by LDC teachers performed differently on the ELA assessment than did their 
matched peers in the comparison group. 


Table 7.16 


Effect Estimates of Cohort 1 Middle School Teachers Joining LDC in 2017-2018 on 2017-2018 New York 
State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Dosage-independent model 
coefficient (SD) 


Variables Dosage-dependent model 
coefficient (SD) 


Level 2 - LDC teacher treatment -0.120 (0.204) -0.061 (0.077) 


Level 1 - student characteristics 


Hispanic 

Black 

Asian 

Other 

Poverty 

Female 

English language learner 
Special education 

Grade 5 at baseline 
Grade 6 at baseline 
Content Exposure Weeks 
Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


-0.060 (0.138) 
-0.167 (0.142) 
0.240 (0.208) 
-0.610 (0.325) 
-0.006 (0.056) 
0.231 (0.031)* 
-0.043 (0.053) 
-0.105 (0.042)* 
0.052 (0.059) 
0.138 (0.062)* 
-0.009 (0.008) 
-0.007 (0.006) 
0.240 (0.046)* 
0.175 (0.026)* 
0.540 (0.027)* 


-0.060 (0.139) 
-0.168 (0.142) 
0.239 (0.207) 
-0.604 (0.323) 
-0.005 (0.056) 
0.231 (0.031)* 
-0.043 (0.053) 
-0.104 (0.041)* 
0.053 (0.059) 
0.142 (0.062)* 
-0.009 (0.008) 
-0.007 (0.006) 
0.240 (0.046)* 
0.174 (0.026)* 
0.540 (0.027)* 


Note. Based on the dosage-dependent model, the average treated student received a 0.281 treatment 
dosage. Because of this, using the dosage dependent model we estimate an average treatment effect on 
the treated (ATET) at (0.281 *- 0.120) = -0.034. 
*Cl two tailed probability > .95 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Prior ELA performance was the strongest 
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predictor and prior mathematics performance also helped explain the outcome. In addition to 
prior achievement, gender and special education status also helped predict the outcome. 


Table 7.17 


Effect Estimates of Cohort 1 Middle School Teachers Who Began Participating in LDC in 2017-2018 on 
2018-2019 New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables 


Dosage-dependent model 


coefficient (SD) 


coefficient (SD) 


Dosage-independent model 


Level 2 - LDC teacher treatment 
Level 1 - student characteristics 
Hispanic 
Black 
Poverty 
Female 
English language learner 
Special education 
Grade 4 at baseline 
Grade 5 at baseline 
Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


0.421 (0.297) 


-0.019 (0.118) 
-0.092 (0.126) 
0.125 (0.064) 
0.231 (0.044)* 
0.174 (0.076)* 
-0.102 (0.065) 
0.078 (0.107) 
0.069 (0.107) 
-0.002 (0.010) 
0.394 (0.075)* 
0.148 (0.038)* 
0.513 (0.039)* 


0.133 (0.139) 


-0.022 (0.118) 
-0.093 (0.126) 
0.125 (0.064) 
0.232 (0.044)* 
0.170 (0.076)* 
-0.101 (0.065) 
0.090 (0.108) 
0.067 (0.108) 
-0.002 (0.010) 
0.392 (0.075)* 
0.149 (0.038)* 
0.513 (0.039)* 


Note. Based on the dosage-dependent model, the average treated student received a 0.355 treatment 
dosage. Because of this, using the dosage dependent model we estimate an average treatment effect on 


the treated (ATET) at (0.355*0.421) = 0.149. 


*Cl two tailed probability > .95 


We also conducted student outcome analysis based on the 2018-2019 New York ELA 


scores for those students taught by Cohort 1 middle school teachers who began participating in 
2017-2018 and continued through 2018-2019 (matching sample sizes and results can be found 
in Tables 113 and 114). As can be seen in Table 7.17, model results for the LDC effect on student 


outcomes are in the positive direction, but are not statistically significant for either model. In 


other words, neither analysis provided sufficient evidence to conclude that students taught by 


LDC teachers performed better on the ELA assessment than did their matched peers in the 


comparison group. 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Prior ELA performance was the strongest 
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predictor and prior mathematics performance also helped explain the outcome. In addition to 
prior achievement, females performed at significantly higher levels than did males. One 
interesting result was that English language learners in this analysis were found to have 
Statistically significant higher ELA scores than did the non-English language learner students. 


7.7 Supplementary Outcome Analysis Results: Cohort 2 Elementary Teachers 


We conducted analyses for Cohort 2 elementary teachers who implemented LDC in 2017— 
2018 and who implemented in both 2017-2018 and 2018-2019. Due to small sample size, no 
analyses were conducted for Cohort 2 elementary teachers who newly joined existing LDC PLCs 
in 2018-2019. 


Table 7.18 


Effect Estimates of Cohort 2 LDC Elementary School Teacher Participating in LDC for 1 year on 2017-2018 
New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Dosage independent 
model coefficient (SD) 


Variables Dosage Dependent 
model coefficient (SD) 


Level 1 - LDC teacher treatment 


Level 2 - Student characteristics 
Hispanic 
Black 
Asian 
Other 
Poverty 
Female 
English Language Learner 
Special education 
Teacher experience 
Baseline peer ELA Z score 
Baseline year mathematics Z score 


Baseline year ELA Z score 


-0.048 (0.066) 


-0.077 (0.073) 
-0.110 (0.074) 
0.015 (0.094) 
-0.028 (0.121) 
-0.056 (0.039) 
0.079 (0.025) * 
-0.160 (0.040) * 
-0.154 (0.036) * 
0.001 (0.004) 
0.100 (0.039) * 
0.235 (0.022) * 
0.593 (0.022) * 


-0.003 (0.034) 


-0.076 (0.073) 
-0.110 (0.074) 
0.015 (0.094) 
-0.029 (0.121) 
-0.056 (0.039) 
0.079 (0.025) * 
-0.161 (0.040) * 
-0.154 (0.036) * 
0.001 (0.004) 
0.099 (0.039) * 
0.234 (0.022) * 
0.593 (0.022) * 


Note. All students were in Grade 3 at baseline year. Based on the dosage-dependent model, the average 
treated student received a 0.642 treatment dosage. Because of this, using the dosage-dependent model 
we could estimate an average treatment effect on the treated (ATET) at (0.642 * -0.048) = -0.031. 

*Cl two tailed probability > .95 


Table 7.18 presents results of both the dosage dependent and dosage independent models on 
Cohort 2 elementary school students’ ELA performance in 2017-2018 after being taught by 
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teachers with 1 year of LDC experience (matching sample sizes and results can be found in 
Tables 116 and 117). The LDC effect on student outcomes are in the negative direction, but are 
not statistically significant for either model. In other words, neither analysis provided sufficient 
evidence to conclude that students taught by LDC teachers performed at levels different than 
their matched peers in the comparison group. 


The effects of the covariates on student performance were in the expected directions. 
Female students performed at higher levels than male students, English Language Learners 
performed at lower levels than non-ELL students, and students classified as special education 
performed at lower levels than did students not classified as special education. 


Table 7.19 


Effect Estimates of Cohort 2 Elementary School Teachers Participating in LDC in Two Consecutive Years 
on 2018-2019 Smarter Balanced ELA Performance, Dosage-Dependent and Dosage-Independent Models 


iy aarsniess Dosage-dependent model Dosage-independent model 
coefficient (SD) coefficient (SD) 
Level 2 LDC teacher treatment 0.094 (0.138) 0.058 (0.094) 


Level 1 student characteristics 


Hispanic -0.038 (0.131) -0.039 (0.131) 
Black -0.165 (0.135) -0.166 (0.135) 
Other 0.574 (0.403) 0.568 (0.403) 
Poverty -0.043 (0.086) -0.042 (0.085) 
Female 0.130 (0.054)* 0.131 (0.054)* 


English language learner 

Special education 

Teacher Experience 

Baseline peer ELA Z score 

Baseline year mathematics Z score 


Baseline year ELA Z score 


0.079 (0.082) 
-0.334 (0.079)* 
0.011 (0.007) 
0.121 (0.084) 
0.226 (0.044)* 
0.559 (0.048)* 


0.079 (0.082) 
-0.335 (0.079)* 
0.012 (0.007) 
0.121 (0.084) 
0.227 (0.043)* 
0.559 (0.048)* 


Note. All students were in Grade 3 at baseline year. Based on the dosage-dependent model, the average 
treated student received a 0.591 treatment dosage. Because of this, using the dosage-dependent model 
we could estimate an average treatment effect on the treated (ATET) at (0.591 * 0.094) = 0.056. 

*Cl two tailed probability > .95 


We next present results of both the dosage dependent and dosage independent models 
on Cohort 2 elementary school students’ ELA performance in 2018-2019 after being taught by 
teachers with 2 years of LDC experience. Matching sample sizes and results can be found in 
Tables 119 and 120. As can be seen in Table 7.19, model results for the LDC effect on elementary 
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student outcomes are in the positive direction, but are not statistically significant for either 
model. In other words, neither model provided sufficient evidence to conclude that elementary 
students taught by LDC teachers performed better on the ELA assessment than did their 
matched peers in the comparison group. 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected direction. Baseline ELA performance was the strongest 
predictor and baseline mathematics performance also helped explain the outcome. In addition 
to baseline achievement, two demographic variables helped predict performance: 1) females 
performed at significantly higher levels than males, and 2) special education students 
performed at lower levels than did students not in special education. 


7.8 Supplementary Outcome Analysis Results: Cohort 2 Middle School Teachers 


We conducted analyses for Cohort 2 middle school teachers who implemented LDC in 
2017-2018, who implemented in both 2017-2018 and 2018-2019, and who newly joined and 
implemented LDC in 2018-2019. In Table 7.20 we present results for the students taught by 
Cohort 2 middle school teachers participating in LDC in 2017-2018 (matching sample sizes and 
results can be found in Tables 122 and 123). Both model results for the LDC effect on student 
outcomes are in the positive direction and did not yield a statistically significant LDC effect. 
Once again, covariates were in the expected direction, with in this case, gender, English 
Language Learner status, and special education status being significant predictors of ELA 
achievement. 
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Table 7.20 


Effect Estimates for Cohort 2 LDC Middle School Teachers Participating in LDC for 1 year on 2017-2018 
New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables Dosage-dependent model Dosage-independent model 
coefficient (SD) coefficient (SD) 
Variables Dosage-dependent model Dosage-independent model 
coefficient (SD) coefficient (SD) 
Level 2 - LDC teacher treatment 0.202 (0.108) 0.078 (0.051) 


Level 1 - student characteristics 


Hispanic 0.107 (0.067) 0.107 (0.067) 
Black 0.021 (0.066) 0.021 (0.066) 
Asian 0.113 (0.074) 0.112 (0.074) 
Other -0.086 (0.116) -0.088 (0.116) 
Poverty -0.056 (0.032) -0.055 (0.032) 
Female 0.130 (0.024)* 0.130 (0.024)* 
English language learner -0.195 (0.044)* -0.196 (0.044)* 
Special education -0.167 (0.032)* -0.167 (0.032)* 
Grade 5 at baseline 0.039 (0.043) 0.037 (0.043) 
Grade 6 at baseline 0.080 (0.039)* 0.081 (0.040)* 
Teacher Experience 0.003 (0.004) 0.003 (0.004) 
Baseline Peer ELA Z score 0.092 (0.037)* 0.096 (0.037)* 
Baseline year mathematics Z score 0.206 (0.020)* 0.206 (0.019)* 


Note. For the dosage dependent model, since the average treatment student received a 0.400 
treatment dosage we could estimate an average treatment on the treated (ATET) at 0.40 * 0.202 = 
0.081. 

*Cl two tailed probability > .95 


In Table 7.21 we present results of models that are both dosage dependent and dosage 
independent for teachers who participated in LDC for 2 years: 2017-2018 and 2018-2019 
(matching sample sizes and results can be found in Tables 125 and 126). Dosage dependent 
model results indicate an LDC effect in the negative direction and the dosage independent 
model results was in the positive direction; both effects were very close to zero and not 
Statistically significant. Once again, covariates were in the expected direction, with in this case, 
Black, female, and special education status being significant predictors of ELA achievement. 
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Table 7.21 


Effect Estimates of Cohort 2 Middle School Teachers Participating in LDC in Two Consecutive Years on 


2018-2019 New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables 


Dosage-dependent model 


coefficient (SD) 


Dosage-independent model 


coefficient (SD) 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 

Black 

Other 

Poverty 

Female 

English language learner 
Special education 

Grade 5 at baseline 

Grade 6 at baseline 

Teacher Experience 


Baseline peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


-0.004 (0.135) 


-0.049 (0.052) 
-0.130 (0.054) * 
0.086 (0.129) 
-0.070 (0.039) 
0.165 (0.030)* 
-0.030 (0.051) 
-0.112 (0.040)* 
-0.035 (0.047) 
0.064 (0.048) 
0.000 (0.004) 
0.321 (0.039)* 
0.179 (0.024)* 
0.537 (0.026)* 


0.061 (0.053) 


-0.051 (0.052) 
-0.131 (0.054) * 
0.083 (0.129) 
-0.070 (0.038) 
0.165 (0.030)* 
-0.030 (0.051) 
-0.112 (0.040)* 
-0.034 (0.046) 
0.065 (0.048) 
0.000 (0.004) 
0.322 (0.039)* 
0.179 (0.024)* 
0.536 (0.026)* 


Note. For the analysis here, Asian & White were combined as reference category to provide adequate 
sample, and the variable of total content weeks was not included as it did not reach our tolerance 
criteria. Based on the dosage-dependent model, the average treated student received a 0.336 treatment 
dosage. Because of this, using the dosage-dependent model we could estimate an average treatment 
effect on the treated (ATET) at (0.336 * 0.004) = -0.001. 

*Cl two tailed probability > .95 


In Table 7.22 we present results of models that are both dosage dependent and dosage 
independent for the Cohort 2 middle school teachers who newly joined and participated in LDC 
in 2018-2019 (matching sample sizes and results can be found in Tables 128 and 129). Dosage 
dependent model results indicate an LDC effect in a positive direction that approached but did 
not obtain statistical significance on the student outcome. The dosage independent model was 
in the negative direction and did not yield a statistically significant LDC effect. Once again, 
covariates were in the expected direction, with in this case, Black, female, and special 
education status being significant predictors of ELA achievement. 
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Table 7.22 


Effect Estimates for Cohort 2 LDC Middle School Teachers Joining LDC in 2018-2019 on 2018-2019 New 
York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables Dosage-dependent model Dosage-independent model 


coefficient (SD) 


coefficient (SD) 


Level 2 - LDC teacher treatment 
Level 1 - student characteristics 
Hispanic 
Black 
Other 
Poverty 
Female 
English language learner 
Special education 
Grade 5 at baseline 
Grade 6 at baseline 
Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


0.029 (0.173) 


-0.151 (0.087) 
-0.224 (0.084)* 
-0.283 (0.192) 
-0.088 (0.053) 
0.186 (0.040)* 
0.048 (0.068) 
-0.125 (0.052)* 
0.155 (0.058)* 
0.189 (0.065)* 
0.013 (0.006)* 
0.233 (0.053)* 
0.183 (0.032)* 
0.556 (0.035)* 


-0.023 (0.067) 


-0.151 (0.087) 
-0.224 (0.084)* 
-0.284 (0.192) 
-0.087 (0.053) 
0.186 (0.040)* 
0.050 (0.068) 
-0.126 (0.052)* 
0.153 (0.058)* 
0.187 (0.065)* 
0.013 (0.006)* 
0.232 (0.053)* 
0.182 (0.032)* 
0.557 (0.035)* 


Note. For the dosage dependent model, since the average treatment student received a 0.323 


treatment dosage we could estimate an average treatment on the treated (ATET) at 0323 * 0.029 = 


0.009. 
*Cl two tailed probability > .95 


7.9 Supplementary Outcome Analysis Results: Prior Year and Outcome Year 


Exposure Subgroups 


In this section, we present results exploring sub-groups of teachers and students from the 


Cohorts 1 and 2 middle school sample. These analyses explore whether the positive results 
found for the overall Cohorts 1 and 2 middle school sample are being driven by subgroups of 
teachers and/or students. 


First, we explore whether students with sustained exposure to LDC over time performed 
at relatively higher levels than did students exposed to LDC for the first time in the primary 
outcome year. To answer this question, we constructed two mutually exclusive subgroups of 
students: students who were exposed to LDC in 2 consecutive years (2016-2017 and 2017- 
2018 for Cohort 1; 2017-2018 and 2018-2019 for Cohort 2), and students who were only 
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exposed to LDC in the second year (2017-2018 for Cohort 1; 2018-2019 for Cohort 2). Matched 
control groups were selected for each subgroup of students. The full results of the model can 
be found in Table 131 in Appendix |. The model includes indicator variables for the two 
treatment groups and the matched control group for the students exposed to LDC in 

2 consecutive years. The control group for the students exposed to LDC in just the outcome 
year was used as the reference group in the model. 


Our primary substantive interest, however, is not in the coefficients as displayed in 
Table 1.31, but rather in two post-hoc tests of the statistical difference between coefficients, 
displayed in Table 7.23. Using the MLwiN software’, we obtained the Chi Square value 
(1 degree of freedom) for a test of the difference between two coefficients. The first test 
examines the difference between the coefficient for the treatment group of students exposed 
to LDC in 2 consecutive years, and the coefficient for the control group selected specifically for 
these treatment students. Although the difference (0.046) is in the expected direction, with the 
treatment group coefficient higher than the control group coefficient, the difference (with a 
p value of 0.274) is not significant. The second test examines the difference between the two 
treatment groups: 1) those students with 2 years of LDC exposure and 2) those students with 
only exposure in the outcome year. The difference (0.102) is in the expected direction and 
statistically significant (p = 0.008). Students with 2 years of LDC experience scored significantly 
higher than the students with only 1 year of LDC experience. 


Table 7.23 
Subgroup results examining exposure of students to LDC over time 


Difference test Difference x p value 


Difference between coefficient for students exposed to LDC for two 0.046 1.201 0.273 
consecutive year and coefficient for matched control student group 


Difference between coefficient for students exposed to LDC for two 0.102 6.986 0.008 
consecutive year and coefficient for students with exposure to LDC 
only in the outcome year 


Next, we explore whether the impact for students with greater exposure to LDC teachers 
is different from the impact for students with less exposure to LDC teachers. Again, we 
constructed two mutually exclusive treatment groups: 1) a high dosage group consisting of 
students who were exposed to LDC in half or more of their core content area class time, and 2) 
a low dosage group consisting of students who were exposed to LDC in less than half of their 
core content area class time. Like the model for which results are displayed in Tables 7.23 and 
Table 131, this analysis selected matched control groups for each of the two mutually exclusive 


4 See http://www. bristol.ac.uk/cmm/software/mlwin/ 
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treatment groups, and indicators for three of the four groups that were included in the model. 
Full model results are displayed in Table 133, and the difference tests of interest are displayed in 
Table 7.24. As can be seen, the coefficient for the high dosage treatment group is smaller than 
the coefficients for both the high dosage matched control group and the low dosage treatment 
group, and these differences (-0.036 and -0.044 respectively) are not statistically significant. 


Table 7.24 


Subgroup results examining high and low LDC dosage in the outcome year 


Difference test Difference xX P value 


Difference between coefficient for students exposed to LDC in -0.036 0.295 0.587 
over half of their core content class time and coefficient for 
matched control student group 


Difference between coefficient for students exposed to LDC in -0.044 0.634 0.426 
over half of their core content class time and coefficient for 

students exposed to LDC in less than half of their core content 

class time 


7.10 Summary and Interpretation of Results 


Here we summarize the student outcome analysis results. Table 7.25 includes the results 
of all of our analyses of the impact of LDC as practiced by teachers with 2 or more years of 
experience with program. In our previous report (Wang et al., 2019), we shared results of the 
impact of Cohort 1 middle school teachers with 2 years of LDC participation. That analysis did 
not identify a statistically significant effect, and a parallel analysis on Cohort 1 elementary 
school teachers was not possible due to insufficient sample size. The sample size for elementary 
sample is usually smaller than the sample size for the middle school sample as expected. In this 
report, we share for the first time the results of the impact of Cohort 2 teachers alone and the 
pooled group of Cohort 1 and Cohort 2 teachers. We also share the results of an analysis 
looking at the impact of Cohort 1 middle school teachers with 3 consecutive years of LDC 
experience, and an analysis of the impact of Cohort 1 middle school teachers who began 
implementing in 2017-2018 (the second year of school implementation). 


Although effect estimates tended to be in the positive direction, none were statistically 
significant, and therefore these analyses do not provide evidence that students exposed to LDC 
instruction as taught by teachers with 2 or more years of experience performed at different 
levels on the New York State ELA Assessment than matched comparison students. 
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Table 7.25 


Impact of LDC on New York State ELA Assessment Scores by Cohort and School Level 


Cohort/teachers 


LDC analytical sample 
information 


Dosage- 
dependent 
model 
coefficient (SD) 


Dosage- 
independent 
model 
coefficient (SD) 


Cohort 1 elementary schools/ 
teachers with 2 years of LDC (starting 
2016-2017) 


Cohort 1 middle schools/teachers 
with 2 years of LDC (starting 2016- 
2017) 


Cohort 1 middle schools/teachers 
with 2 years of LDC (starting 2017- 
2018) 


Cohort 2 elementary schools/ 
teachers with 2 years of LDC (starting 
2017-2018) 


Cohort 2 middle schools/teachers 
with 2 years of LDC (starting 2017- 
2018) 


Cohorts 1 & 2 elementary schools/ 
teachers with 2 years of LDC (starting 
in 2016-2017 for Cohort 1 and 2017- 
2018 for Cohort 2) 


Cohorts 1 & 2 middle schools/ 
teachers with 2 years of LDC (starting 
in 2016-2017 for Cohort 1 and 2017- 
2018 for Cohort 2) 


Cohorts 1 & 2 elementary & middle 
schools/teachers with 2 years of LDC 
(starting in 2016-2017 for Cohort 1 
and 2017-2018 for Cohort 2) 


Cohort 1 middle schools/teachers 
with 3 years of LDC (starting 2016- 
2017) 


2 schools / 2 teachers / 
35 students? 


10 schools / 26 
teachers / 1,408 
students 


2 schools / 8 teachers / 
500 students 


9 schools / 17 teachers 
/ 341 students 


9 schools / 22 teachers 
/ 947 students 


11 schools / 19 
teachers / 363 students 


19 schools / 48 
teachers / 2,355 
students 


29 schools / 66 
teachers / 2,718 
students 


6 schools / 11 teachers 
/ 406 students 


Not conducted 


0.001 (0.133) 


0.421 (0.297) 


0.094 (0.138) 


-0.004 (0.135) 


0.041 (0.139 


0.005 (0.092) 


0.019 (0.075) 


0.125 (0.250) 


Not conducted 


-0.009 (0.056) 


0.133 (0.139) 


0.058 (0.094) 


0.061 (0.053) 


0.025 (0.091) 


0.020 (0.038) 


0.020 (0.035) 


0.010 (0.090) 


Note. *These are the numbers before conducting matching analysis. 


*CI two tailed probability > .95 


Next, we provide a lens through which the reader can contextualize the magnitude of the 
results. We present dosage-dependent effects for each of the three primary analyses based on 
teachers with 2 years of LDC implementation. Figure 7.1 depicts the estimated impacts of LDC 
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in the three samples of students exposed to LDC teachers in all three major content areas: ELA, 
social studies/history, and science. These effect sizes can be best understood as the estimated 
impact of LDC under ideal conditions. Figure 7.2 depicts the estimated impact of LDC in the 
three samples on the average observed student, who in the middle school context had 
considerably less exposure to LDC teachers in his or her core content classes. 


As can be seen in the figures, the confidence intervals all cross the zero line, and therefore 
the estimates are not statistically significant at the 95 percent level. Higher than expected levels 
of school and teacher attrition, and several changes in LDC project leadership are likely to have 
contributed to this lack of statistical significance. The analyses provide no evidence of an effect 
on students taught by either elementary or middle school teachers who implemented LDC for 2 
years. 


Figure 7.1 


Treatment Effect on New York State Assessment ELA Scores with 95% Confidence Interval for Students 
with Full LDC Dosage, by Cohort 


0.50 - 
0.25 - 
Teacher Cohorts 
i [3 Elementary Teachers (n=19) 
S Middle Teachers (n=47) 
= Elementary and Middle Teachers (n=66) 
-0.25 - 
-0.50 - 
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Figure 7.2 


Treatment Effect on New York State Assessment ELA Scores with 95% Confidence Interval for Students 
with Average LDC Dosage, by Cohort 
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8.0 Summary of Findings 


This final annual report examines LDC following 3 years of implementation in Cohort 1 
schools and 2 years of implementation in Cohort 2 schools. We summarize these results 
organized by the three categories of evaluation questions listed in Chapter 1 of this report. 


8.1 Program Characteristics and Implementation 


The study schools serve largely Hispanic and Black populations, with a high proportion of 
students from economically disadvantage backgrounds, and many English language learners. 
Consistent with prior year results, participants reported very positive attitudes toward LDC and 
its implementation at their schools. Teachers generally appreciated the opportunity to 
collaborate with colleagues, and nearly uniformly praised their LDC coaches. 


Evidence suggests that implementation was largely faithful to the intended structure of 
LDC, although the level of implementation was not uniformly high. The frequency of PLC 
meetings varied greatly across schools, with some struggling to meet frequently and ensure 
high attendance among teachers. On a positive note, PLC members were nearly uniform in 
reporting that their teacher leaders were supportive, knowledgeable, and helpful. Teacher 
leaders themselves also reported high satisfaction with support from coaches, professional 
development offerings, and how the teacher leader role allowed them to be instructional 
leaders in their schools. 


Analysis of program data suggests that nearly all participants were engaging with the 
module building platform at least to some degree, but that engagement varied considerably. 
Nearly half of teachers failed to engage in module design at a basic level by editing the teaching 
task, while others engaged deeply by editing multiple module elements. When coaches 
provided feedback via commenting on modules and other methods, PLC members generally 
found it useful. Our fidelity analysis, however revealed that almost 20% of modules did not 
receive the program goal of two coach comments. Furthermore, the national peer review 
process was used by few participants. Module analysis suggests that the materials adapted and 
created by PLC members varied in levels of completion, and based on the presence of uploaded 
student work, a sizable number of modules might not have been implemented in the 
classroom. 


8.2 Contextual Factors and Implementation 


Coaches and teacher leaders were almost universally praised by teachers participating in 
LDC. While teacher respondents generally reported that their school administrators were 
supportive of the program, administrators’ level of participation in PLC meetings and their 
observation of LDC instruction varied greatly across the sample. There was substantial attrition 
of schools and teachers from the program across the 3 years, with many school leaders 
deciding that they didn’t have sufficient resources and/or teacher buy-in to sustain the 
program. Most schools that remained in the program at the end of the study, however, 
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indicated that they would continue to use LDC practices and tools, at least to some extent. 
While school personnel generally perceived district leaders as supportive of LDC, some 
educators did not feel that district leaders fully understood the program. 


8.3 Program Impacts 


Based on survey results, LDC was perceived to have positive impacts on a range of both 
teacher practices and student skills. Teachers, in general, felt that LDC had improved their 
instructional planning and pedagogical skill sets and was also helping to promote collaboration 
between teachers. Teachers and administrators also felt that LDC was improving student 
learning across multiple domains. Furthermore, those attitudes around teacher and student 
skills became even more positive over time on average for teachers completing the survey in 
both 2017-2018 and 2018-2019. Analysis of module quality showed improvement from 2016— 
2017 to 2017-2018, but average quality seemed to decrease slightly from 2017-2018 to 2018-— 
2019. 


Although effect estimates tended to be in the positive direction, none of the quasi- 
experimental analyses of the impact of LDC on student learning after 2 or more years of teacher 
participation in LDC PLCs yielded statistically significant results. The study did not provide 
evidence that students exposed to LDC instruction performed at different levels than did 
matched comparison students on the New York State ELA Assessment. Supplementary analyses 
revealed that students exposed to LDC across 2 consecutive years performed at higher levels 
than students exposed to LDC in just 1 year, suggesting that sustained exposure to LDC over 
time may yield greater benefits to students than exposure in 1 year alone. 
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Appendix A: 
Teacher Survey and Responses 


T1. 


T2a. 


T2b. 


T2c. 


T2d. 


T3. 


2018-2019 
LDC Teacher Survey 


LDC Participation 


Please select your school from the drop down box. 
Teachers are skipped to T3 if they teach in an elementary school. 


In the current school year (2018-19), how many classes did you teach? 


| | classes n= 121, Mean = 3.90, Range: 0 - 14 


In how many of these classes did you use LDC modules and/or mini-tasks? 


| | classes n=121, Mean = 2.35, Range: 0- 8 


In what content areas did you use LDC modules and/or mini-tasks? 


In what grades did you use LDC modules and/or mini-tasks? 


Prior to the current school year (2018-19), did you have any experience with LDC? 


(n = 206) 
Yes 124 teachers (60.2%) 
No <> Skip to T5a 82 teachers (39.8%) 
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T4. How many of the following did you teach prior to the current school year (2018-19)? 


| | LDC modules n=124, Mean = 2.69, Range: 0 - 15 


| | LDC mini-tasks, outside of modules n= 124, Mean =3.39, Range: 0 - 20 


Professional Learning Community and Teacher Collaboration 


T5a. Did you participate this year in a Professional Learning Community (PLC) at least partly 
focused on implementing LDC in your school? 


(n = 182) 
Yes y Skip to T6 164 teachers (90.1%) 
No 18 teachers (9.9%) 


T5b. Did you use any LDC tools in your instructional planning or classroom instruction this 


year? 
(n = 18) 
Yes y Skip to 5d 18 teachers (100.0%) 
No 0 teachers (0.0%) 


T5c. Why did you choose not to use any LDC tools in your instructional planning or 
classroom instruction this year? 


[Survey ends here for respondents answering question T5c] 
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T5d. What LDC tools did you use during the current school year? Select all that apply. 


(n= 17) 
CoreTools online platform to access existing modules or mini-tasks 13 teachers (6.3%) 
CoreTools online platform to design modules or mini-tasks 10 teachers (4.9%) 
LDC online courses 6 teachers (2.9%) 
Modules or mini-tasks given to me by other teachers in my school 7 teachers (3.4%) 
Other (please specify) 1 teacher (0.5%) 


[Survey ends here for respondents answering question T5d] 


T6. About how often did your LDC PLC meet? 


(n = 188) 
Less than once a month 12 teachers (6.4%) 
Once a month 33 teachers (17.6%) 
Every other week 39 teachers (20.7%) 
Once a week y SkiptoT8 94 teachers (50.0%) 
Twice a week or more often y SkiptoT8 10 teachers (5.3%) 


T7. What were the primary barriers preventing your LDC PLC from meeting weekly? Select 


all that apply. 


PLC time was not protected. 


PLC members had limited interest in attending meetings. 


School administrator did not make it a priority. 


Teacher Leader did not provide sufficient organizational support. 


Not enough teachers participated. 


Other (please specify) 


PLC members had other priorities that compete with LDC participation. 


(n =84) 

18 teachers (21.4%) 
7 teachers (8.3%) 
12 teachers (14.3%) 
1 teacher (1.2%) 

6 teachers (7.1%) 
49 teachers (58.3%) 
24 teachers (28.6%) 
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T8. About how often did you have informal discussions (as opposed to scheduled meetings) 
about LDC with teachers in your LDC PLC? 


(n = 188) 
Less than once a month 29 teachers (15.4%) 
Once a month 27 teachers (14.4%) 
Every other week 28 teachers (14.9%) 
Once a week 69 teachers (36.7%) 
Twice a week or more 35 teachers (18.6%) 


T9. On average, how long did your school’s LDC PLC meetings typically last? 


(n = 188) 
Less than 45 minutes 32 teachers (17.0%) 
45 to 59 minutes 125 teachers (66.5%) 
60 to 74 minutes 20 teachers (10.6%) 
75 minutes or more 11 teachers (5.9%) 


Teacher Training and Support 


T10. How effective was your LDC PLC in the following areas? 


Not A little Moderately Very 
effective effective effective effective 
Creating an environment in which teachers were 4 10 72 101 
comfortable working together (n = 187) (2.1%) (5.3%) (38.5%) (54.0%) 
Gee ee 3 5 65 108 
. (1.6%) (7.0%) (34.8%) (56.7%) 
(n = 187) 
3 15 66 103 
Allowi to sh tudent k (n = 187 
owing space to share student work (n ) (1.6%) (8.0%) (35.3%) (55.1%) 
Helping teachers to improve their LDC 4 23 69 91 
instructional plans (n = 187) (2.1%) (12.3%) (36.9%) (48.7%) 
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T11. How would you rate each of the following aspects of the online course material (in the 


LEARN tab in LDC CoreTools) that your coach used or directed you to use? 


Poor Fair Good Excellent 
‘ ; ; 2 6 28 87 66 
Clarity of information presented (n = 187) (3.2%) (15.0%) (46.5%) (35.3%) 
; ; - 4 28 94 61 
Relevance of information presented (n = 187) (2.1%) (15.0%) (50.3%) (32.6%) 
| 44 89 45 
E f = 187 
seer Seay ante (4.8%) (23.5%) (47.6%) (24.1%) 
Usefulness of resource documents (e.g., LDC 
8 36 81 62 
Pitfall Checklist, CCSS Mental Markers, etc. 
ee a ce aD te A (4.3%) (19.3%) (43.3%) (33.2%) 
(n = 187) 
Usefulness of videos (n = 187) eo 7 ee . 
- (7.0%) (30.5%) (39.0%) (23.5%) 
Degree to which course material helped teachers 10 34 90 53 
to create and/or adapt LDC modules (n = 187) (5.3%) (18.2%) (48.1%) (28.3%) 
Opportunity to extend learning when needed or 7 31 90 59 
desired (n = 187) (3.7%) (16.6%) (48.1%) (31.6%) 


T12a. Overall, were you able to get the feedback and support you needed from your LDC coach 
(through written feedback in LDC CoreTools, or coaching and modeling in your LDC PLCs) to 
plan, teach, reflect on, and revise LDC modules)? 


(n = 187) 
Yes 174 teachers (93.0%) 
No 13 teachers (7.0%) 


T12b. Did your LDC coach provide written feedback on your module(s) in LDC CoreTools in a 


timely manner? 


(n = 187) 
Yes 179 teachers (95.7%) 
No 8 teachers (4.3%) 
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T13. Outside of the PLC meetings with your LDC coach, please indicate whether you used 
each of the following types of coach support, and how helpful you found these types 


of support. 
Used 
Did not 
use Not A little Moderately Very 

helpful helpful helpful helpful 
Written feedback in LDC CoreTools from 
your LDC coach (in the comments 7 8 29 64 79 
areas and/or via the teacher work rubric) (3.7%) (4.3%) (15.5%) (34.2%) (42.2%) 
(n = 187) 
One-on-one Zoom video conference and/or 37 F 20 55 68 
call with your LDC coach (n = 187) (19.8%) (3.7%) (10.7%) (29.4%) (36.4%) 
Email or phone communication with your 36 3 25 42 81 
LDC coach (n = 187) (19.3%) (1.6%) (13.4%) (22.5%) (43.3%) 
Other (n = 187) 118 4 5 20 40 
Please specify: (63.1%) (2.1%) (2.7%) (10.7%) (21.4%) 


Module Creation 


714. During the current school year (2018-19), how many LDC modules did you individually 
or collaboratively adapt from existing modules (e.g., modules you created in a prior year 
and/or modules found in the LDC Library in CoreTools)? 


| | Adapted modules 


n= 163, Mean = 2.18, Range: 0 - 16 


T15. During the current school year (2018-19), how many LDC modules did you create, either 
individually or with colleague(s)? Only include modules built from scratch, not those 


adapted from existing modules in the LDC library. 


| | New modules 


n= 163, Mean = 1.43, Range: 0-8 
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T16. How did members of your PLC collaborate to create LDC modules? 
Check all that apply. 


(n = 187) 

74 teachers (35.9%) 
120 teachers (58.3%) 
44 teachers (21.4%) 


Modules were created by individual teachers. 


Modules were created by teams of two or more teachers. 


Modules were created by the PLC as a whole. 


Other (please specify) 18 teachers (8.7%) 


T17. Please indicate to what extent you were able to do each of the following when creating 


LDC modules. 


Not at all mints oo Agtest 
extent extent 
Select focus standards for a writing 2 14 61 108 
assignment (n = 185) (1.1%) (7.6%) (33.0%) (58.4%) 
Create a standards-driven writing 2 12 69 102 
assignment (n = 185) (1.1%) (6.5%) (37.3%) (55.1%) 
Select high quality, complex texts and other 0 A 78 33 
materials to engage students in deeper 
learning (n = 185) (0.0%) (13.0%) (42.2%) (44.9%) 
Identify the skills students need to develop 0 15 68 102 
to complete a writing assignment (n = 185) (0.0%) (8.1%) (36.8%) (55.1%) 
Create daily lessons to teach the skills a 3 20 85 77 
student needs to complete a writing 
assignment (n = 185) (1.6%) (10.8%) (45.9%) (41.6%) 
Differentiate instruction by incorporating 
multiple ways of thinking, various levels of Z 34 86 63 
complexity, and multiple modalities (1.1%) (18.4%) (46.5%) (34.1%) 
(n = 185) 
Plan for a variety of methods to assess 5 29 83 71 
student progress (e.g., rubrics and/or mini- 
task scoring guides) (n = 185) (1.1%) (15.7%) (44.9%) (38.4%) 
Assess the quality of writing assignments 
and/or instructional plans using Peer 
Review/Curriculum Alignment Rubric (e.g. 5 ef Pye Phi rae 
Task Pitfalls Checklist, rubric indicators) (2.2%) (8.6%) (45.9%) (43.2%) 
(n = 185) 
Make a writing assignment relevant and 
engaging for students (n = 185) s 2? oe #2 
(0.5%) (12.4%) (37.3%) (49.7%) 
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Classroom Implementation 


T18. How many total LDC modules did you teach during the current school year (2018-19)? 


|| Modules n= 183, Mean = 2.75, Range: 0-15 


T19. Outside of modules, approximately how many individual LDC mini-tasks did you teach during 
the current school year (2018-19)? 


| | Mini-tasks n= 183, Mean = 4.20, Range: 0 - 48 


T20. Please indicate to what extent you were able to do each of the following activities when 
teaching LDC modules. 


Not at all AlRebR ON ooo Melee 
extent extent 

Engage students in understanding the 4 16 77 86 
assignment and its rubric (n = 183) (2.2%) (8.7%) (42.1%) (47.0%) 
Engage students in accessing complex text 2 15 84 82 
for the purpose of the assignment (n = 183) (1.1%) (8.2%) (45.9%) (44.8%) 
Systematically collect information about 3 26 82 7Z 
students’ progress (n = 183) (1.6%) (14.2%) (44.8%) (39.3%) 
Provide feedback to students using 3 17 73 85 
assignment rubrics (n = 183) (1.6%) (9.3%) (42.6%) (46.4%) 
Locate evidence of standards in final 3 17 78 85 
student work on the writing assignment (1.6%) (9.3%) (42.6%) (46.4%) 
(n = 183) 
Use evidence of student progress on 4 1 72 86 
standards to modify subsequent instruction ‘ P F ‘ 
(n = 183) (2.2%) (11.5%) (39.3%) (47.0%) 
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T21. Toward the beginning of the school year, did you “find and teach” a module from CoreTools? 


(n = 183) 
Yes 110 teachers (60.1%) 
No Jt Skip to T24 73 teachers (39.9%) 


T22. What was the name of the Find and Teach module? 


T23. Did you make any adjustments to the Find and Teach module? 


(n = 110) 
Yes 81 teachers (73.6%) 
No 29 teachers (26.4%) 


T24. What module did you adapt, refine, and/or develop most during the current school year 
(2018-19)? This module is typically one you worked on after the Find and Teach module. 


T25. Which of these statements best describes how you created the module named in the previous 
question? 
(n = 182) 
| created a module from a template in CoreTools. 77 teachers (42.3%) 


| found and adjusted another teacher’s module from 105 teachers (57.7%) 
the LDC Library in CoreTools. 


T26. Did you teach this module in your classroom? 
(n = 182) 
Yes, | have already taught this module this year. 159 teachers (87.4%) 


No, but | plan to teach this module before the end of the 2018-19 school year. 11 teachers (6.0%) 


No, but I plan to teach this module during next school year. 5 teachers (2.7%) 


No. | do not currently have plans to teach this module in my classroom. 7 teachers (3.8%) 
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Module Peer Review 


T27. Did you attend a Peer Review/Curriculum Alignment Workshop this school year? (Y/N) 


(n = 182) 
Yes 52 teachers (28.6%) 
No 130 teachers (71.4%) 


T28. How many modules did you submit online for LDC National Peer Review during the 
current school year (2018-19)? 


| | Modules |f none, GY siipto T29 n = 182, Mean = 0.81, Range: 0-4 


T29. How helpful did you find the National Peer Review process in improving the quality of 
your module? 


(n = 117) 
Not helpful 4 teachers (4.5%) 
A little helpful 29 teachers (32.6%) 
Moderately helpful 30 teachers (33.7%) 
Very helpful 26 teachers (29.2%) 
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7 Impact on Teacher Practice and Learning 


T30. Between the beginning and end of this year’s work with LDC, please indicate how 
much your skills have improved in the following areas: 


Not at all A little Moderately | A great deal 
Selecting focus standards for a writing assignment 7 20 74 82 
(n = 183) (3.8%) (10.9%) (40.4%) (44.8%) 
Creating standards-driven writing assignments 4 24 71 84 
(n = 183) (2.2%) (13.1%) (38.8%) (45.9%) 
Identifying the skills students need to develop to 6 22 75 80 
complete a writing assignment (n = 183) (3.3%) (12.0%) (41.0%) (43.7%) 
Creating daily lessons to teach the skills students need 9 26 75 73 
to complete a writing assignment (n = 183) (4.9%) (14.2%) (41.0%) (39.9%) 
Systematically collecting information on students’ 9 29 75 70 
progress (n = 183) (4.9%) (15.8%) (41.0%) (38.3%) 
Identifying patterns of student understandings or | 24 76 74 
misconceptions (n = 183) (4.9%) (13.1%) (41.5%) (40.4%) 
Using evidence of student progress on standards to i) 28 Pic ve 
modify subsequent instruction (n = 183) (2.7%) (15.3%) (41.0%) (41.0%) 
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T31. Please indicate the degree to which you agree or disagree with the statements below. 


Stonely Disagree Agree Strongly 

disagree agree 
Participating in LDC raised my expectations for 6 16 99 62 
students’ writing. (n = 183) (3.3%) (8.7%) (54.1%) (33.9%) 
Using LDC modules became an important part of my g 29 96 49 
instructional practice. (n = 183) (4.9%) (15.8%) (52.5%) (26.8%) 
Implementing LDC helped me incorporate my state’s 8 23 105 AT 
College- and Career-Ready Standards into my AAS 12.6% 57.4% 25.7% 
instruction. (n = 183) (4.4%) (12.6%) (57.4%) (25.7%) 
LDC helped me incorporate writing assignments into i) 20 105 53 
my existing curriculum. (n = 183) (2.7%) (10.9%) (57.4%) (29.0%) 
Sie eee nae || gs . sy - 
Professional Learning Community. (n = 183) (3.8%) (13.1%) (55.7%) (27.3%) 
LDC helped me improve on my teacher evaluation 13 34 100 36 
ratings. (n = 183) (7.1%) (18.6%) (54.6%) (19.7%) 
iwi rah eee) ae 26 103 45 
subjects. (n= 183) (4.9%) (14.2%) (56.3%) (24.6%) 
| shared my LDC work with colleagues outside of the 17 46 84 36 
LDC PLC. (n = 183) (9.3%) (25.1%) (45.9%) (19.7%) 


156 


Impact on Student Learning 


T32. Please indicate to what extent LDC had a positive effect on students in the following areas. 


Not at all A little Moderately | A great deal 
11 34 85 53 
Reading skills (n = 183 
eadine swisweoae (6.0%) (18.6%) (46.4%) (29.0%) 
8 27 78 70 
Content knowledge (n = 183) (4.4%) (14.8%) (42.6%) (38.3%) 
or das : 4 ZT 73 79 
Ability to complete writing assignments (n = 183) (2.2%) (14.8%) (39.9%) (43.2%) 
; ye sades 6 LT 75 75 
Quality of students’ writing (n = 183) (3.3%) (14.8%) (41.0%) (41.0%) 
: 12 35 83 53 
College and career ready skills (n = 183) (6.6%) (19.1%) (45.4%) (29.0%) 
Capacity to analyze and understand the 4 34 72 73 
components of a writing assignment (n = 183) (2.2%) (18.6%) (39.3%) (39.9%) 
; : : : _ 8 31 86 58 
Speaking and listening skills (n = 183) (4.4%) (16.9%) (47.0%) (31.7%) 
4 31 93 55 
O I lit f = 183 
verall literacy performance (n ) (2.2%) (16.9%) (50.8%) (30.1%) 
Performance on assessments throughout the 10 30 81 62 
school year (n = 183) (5.5%) (16.4%) (44.3%) (33.9%) 
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Teacher Leader Support 


The following question refers to the LDC project liaison in your school. This is the teacher leading your 


Professional Learning Community work. 


T33. Please indicate the degree to which you agree or disagree with the following statements. 


stronely Disagree Agree Strongly 

disagree agree 
Our school’s LDC teacher leader effectively 
supported our Professional Learning Community a ‘ ie et 
meetings. (n = 159) (0.0%) (4.4%) (44.7%) (50.9%) 
Ase ee eaten : Sees ’ ~ - 
leader. (n = 159) (0.0%) (3.1%) (41.5%) (55.3%) 
Our teacher leader helped teachers align LDC to 0 14 66 79 
broader school instructional goals. (n = 159) (0.0%) (8.8%) (41.5%) (49.7%) 
Our teacher leader offered useful feedback for a 12 66 80 
the design and revision of LDC modules. (n = 159) (0.6%) (7.5%) (41.5%) (50.3%) 
Our teacher leader was effective in inviting 2 8 68 81 
teachers to join the LDC initiative. (n = 159) (1.3%) (5.0%) (42.8%) (50.9%) 


410)) School Administrator Support 


The following questions refer to the school administrator who oversees the LDC project at your school. 


T34. What proportion of PLC meetings focused on LDC did your school administrator attend? 


Less than one quarter of LDC PLCs 


About one quarter of LDC PLCs 
About one half of LDC PLCs 
About three quarters of LDC PLCs 


More than three quarters of LDC PLCs 


(n = 183) 

65 teachers (35.5%) 
41 teachers (22.4%) 
30 teachers (16.4%) 
15 teachers (8.2%) 
32 teachers (17.5%) 
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T35. How many times did your school administrator observe you teach an LDC mini-task 
during the current school year (2018-19)? 


(n = 183) 


0 times 69 teachers (37.7%) 
1 time 45 teachers (24.6%) 
2 times 39 teachers (21.3%) 
3 or more times 30 teachers (16.4%) 


T36. Please indicate the degree to which you agree or disagree with the following statements. 


My school administrator... Stoney Disagree Agree strongly 
disagree agree 
11 23 100 49 

had a fi derstandi f LDC. (n = 183 

a a (6.0%) (12.6%) (54.6%) (26.8%) 
Allocated resources such as teacher time, 
payment, administrator time, support 14 oe 98 60 
staff, sub coverage, etc., to ensure the LDC (7.7%) (6.0%) (53.6%) (32.8%) 
team could meet. (n = 183) 
encouraged teachers to participate in LDC. Z 4 99 73 
(n = 183) (3.8%) (2.2%) (54.1%) (39.9%) 
oe ae a 
instructional priorities (n = 183) (21.9%) (26.8%) (34.4%) (16.9%) 
cpporedspece: Goole eidisr|' g° = 
goals. (n = 183) (6.0%) (10.4%) (53.0%) (30.6%) 
provided me with feedback about my LDC 20 24 101 38 
planning and/or instruction. (n = 183) (10.9%) (13.1%) (55.2%) (20.8%) 
made formative assessment a priority at a 16 104 54 
my school. (n = 183) (4.9%) (8.7%) (56.8%) (29.5%) 
ee eee | a 19 95 55 
: 65) . (7.7%) (10.4%) (51.9%) (30.1%) 
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Teacher Leadership Role 


11 


T37. Please indicate the degree to which you agree or disagree with the following 
statements about your role in your school’s LDC implementation. 


strouely Disagree Agree Strongly 
disagree agree 
| was involved in setting instructional goals for the LDC 14 34 80 30 
work at my school. (n = 158) (8.9%) (21.5%) (50.6%) (19.0%) 
| was involved in discussions about how to expand LDC 15 42 74 27 
implementation at my school in future years. (n = 158) (19.5%) (26.6%) (46.8%) (17.1%) 
| had the opportunity to work with our LDC teacher leader 14 38 75 31 
and our administrator to help shape LDC implementation. : 
(n = 158) (8.9%) (24.1%) (47.5%) (19.6%) 
pee ene eee oci clln inee a 26 73 34 
fo} fo} fo} fo} 
coach, providing feedback to my peers, etc. (n = 158) (15.8%) (16.5%) (46.2%) (21.5%) 


Facilitators and Barriers 
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T38. Please indicate the degree to which you agree or disagree with the following statements. 


Strongly Disagree Agree Sronely 

disagree agree 
My LDC PLC was given sufficient time to meet. 5 36 99 42 
(n = 182) (2.7%) (19.8%) (54.4%) (23.1%) 
| felt adequately prepared to effectively implement 6 21 110 45 
LDC modules in my classroom. (n = 182) (3.3%) (11.5%) (60.4%) (24.7%) 
It was challenging to find content-rich reading 13 50 81 38 
materials for the LDC modules | developed. (n = 182) (7.1%) (27.5%) (44.5%) (20.9%) 
My school had adequate technology to support 9 18 99 56 
teachers’ use of LDC. (n = 182) (4.9%) (9.9%) (54.4%) (30.8%) 
It was easy to find and adapt LDC mini-tasks for use in 4 27 103 48 
my classroom. (n = 182) (2.2%) (14.8%) (56.6%) (26.4%) 
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if6}) Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 


e CoreTools online platform 
e LDC online courses in the “LEARN” section of CoreTools 
e Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e In-person coaching 


o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


T39. What supports did you find the most useful and why? 


T40. What supports were not helpful and why? 


T41. In what ways could LDC implementation be improved in your school in the future? 
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Appendix B: 
Teacher Leader Survey and Responses 


2018-2019 
LDC Teacher Leader Survey 


LDC Participation 


TL1. Prior to the current school year (2018-19), did you have any experience with LDC? 


(n = 30) 
Yes 29 Teacher Leaders (96.7%) 
No y Skip to TL3 1 Teacher Leader (3.3%) 


TL2. How many of the following did you teach prior to the current school year (2018-19)? 


| | LDC modules n= 29, Mean = 2.68, Range: 0-6 


LDC mini-tasks, outside of modules n = 29, Mean = 3.90, Range: 0 - 15 
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Professional Learning Community and Teacher Collaboration 


The following questions involve the LDC Professional Learning Community (PLC) that you are leading. 


TL3. About how often did your LDC PLC meet? 


Less than once a month 


Once a month 


Every other week 


Once a week 


Twice a week or more often 


<> Skip to TLS 
<> Skip to TLS 


(n = 30) 

2 Teacher Leaders (6.7%) 

5 Teacher Leaders (16.7%) 
11 Teacher Leaders (36.7%) 
11 Teacher Leaders (36.7%) 
1 Teacher Leader (3.3%) 
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TL4. What were the primary barriers preventing your LDC PLC from meeting weekly? 


Select all that apply. 


(n = 18) 
PLC time was not protected. 6 Teacher Leaders (33.3%) 
PLC members had limited interest in attending meetings. 2 Teacher Leaders (11.1%) 
School administrator did not make it a priority. 3 Teacher Leaders (16.7%) 
| was unable to provide sufficient organizational support. O Teacher Leaders (0.0%) 
Not enough teachers participated. 2 Teacher Leaders (11.1%) 
PLC members had other priorities that competed with LDC 11 Teacher Leaders (61.1%) 
participation. 
Other (please specify) O Teacher Leaders (0.0%) 


TL5. About how often did you have informal discussions (as opposed to scheduled 
meetings) about LDC with teachers in your LDC PLC? 


Once a week 


Less than once a month 
Once a month 


Every other week 


Twice a week or more 


(n = 30) 

6 Teacher Leaders (20.0%) 
6 Teacher Leaders (20.0%) 
4 Teacher Leaders (13.3%) 
9 Teacher Leaders (30.0%) 
5 Teacher Leaders (16.7%) 


TL6. On average how long did your school’s LDC PLC meetings typically last? 


Less than 45 minutes 
45 to 59 minutes 
60 to 74 minutes 


75 minutes or more 


(n = 30) 

6 Teacher Leaders (20.0%) 
21 Teacher Leaders (70.0%) 
3 Teacher Leaders (10.0%) 
O Teacher Leaders (0.0%) 
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Teacher Training and Support 


TL7. How effective was your LDC PLC in the following areas? 


Not A little Moderately Very 
effective effective effective effective 

Creating an environment in which teachers are 1 (3.3%) 1 (3.3%) 13 (43.3%) | 15 (50.0%) 
comfortable working together (n = 30) 

Fostering an environment where teachers share 1 (3.3%) 2 (6.7%) 12 (40.0%) | 15 (50.0%) 
their instructional plans with colleagues (n = 30) 

Allowing space to share student work (n = 30) 2 (6.7%) 2 (6.7%) 12 (40.0%) | 14 (46.7%) 
Helping teachers learn to improve their LDC 1 (3.3%) 2 (6.7%) 13 (43.3%) | 14 (46.7%) 


instructional plans. (n = 30) 


LEARN tab in LDC CoreTools) that your coach used or directed you to use? 


TL8. How would you rate each of the following aspects of the online course material (in the 


Poor Fair Good Excellent 
Clarity of information presented (n = 30) 0 (0.0%) 3 (10.0%) 15 (50.0%) | 12 (40.0%) 
Relevance of information presented (n = 30) 0 (0.0%) 2 (6.7%) 17 (56.7%) | 11(36.7%) 
Ease of use (n = 30) 0 (0.0%) 6 (20.0%) 17 (56.7%) 7 (23.3%) 
Usefulness of resource documents (e.g., LDC 
Pitfall Checklist, CCSS Mental Markers, etc.) 0 (0.0%) 2 (6.7%) 17 (56.7%) | 11 (36.7%) 
(n = 30) 
Usefulness of videos (n = 30) 0 (0.0%) 10 (33.3%) | 13 (43.3%) 7 (23.3%) 
Degree to which course material helped teachers 0 (0.0%) 4 (13.3%) 17 (56.7%) 9 (30.0%) 
to create and/or adapt LDC modules (n = 30) ie re 7“ ue 
Opportunity to extend learning when needed 0 (0.0%) 4 (13.3%) 16 (53.3%) 10 (33.3%) 


or desired (n = 30) 
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TL9a. Overall, were you able to get the feedback and support you needed from your LDC coach 
(through written feedback in LDC CoreTools, or coaching and modeling in your LDC PLCs) to 
plan, teach, reflect on, and revise LDC modules? 


(n = 30) 


Yes 29 Teacher Leaders (96.7%) 
No 1 Teacher Leader (3.3%) 


TL9b. Did your LDC coach provide written feedback on your module(s) in LDC CoreTools in a 


timely manner? 


(n = 30) 


Yes 29 Teacher Leaders (96.7%) 
No 1 Teacher Leader (3.3%) 


TL10. Outside of the PLC meetings with your LDC coach, please indicate whether you used 
each of the following types of coach support, and how helpful you found these types 


of support. 
Used 
Did not use Not A little Moderately Very 
helpful helpful helpful helpful 

Written feedback in LDC CoreTools 
cattl baby Acetate 2 (6.7%) 2 (6.7%) | 3(10.0%) | 5 (16.7%) | 18 (60.0%) 
comments areas and/or via 
the teacher work rubric) (n = 30) 
One-on-one Zoom video 
conference and/or call with your 2 (6.7%) 1 (3.3%) 0 (0.0%) 8 (26.7%) | 19 (63.3%) 
LDC coach (n = 30) 
Email or phone communication ‘ é F 
With YourlDE coach Wieae) 0 (0.0%) 1 (3.3%) 2 (6.7%) 7 (23.3%) | 20 (66.7%) 
Other ipleasespecity) Gime) 20 (66.7%) | 0(0.0%) | 0(0.0%) | 3(10.0%) | 7 (23.3%) 
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TL11. How many in-person and/or online LDC professional development offerings for school 
administrators and Teacher Leaders did you attend during the current school (e.g., Summer in- 
person launch days, Quarterly in-person Teacher Leader meetings, LDC monthly virtual coach 
meetings)? 


| | Professional development offerings n= 30, Mean = 3.83, Range: 0 - 17 


Support to Teacher Leader from LDC Coach 


TL12. Please indicate the degree to which you agree or disagree with the following statements. 


ree Disagree Agree Sronely N/A 
sagree agree 

| was able to reach my LDC coach if | 

had any questions ee LDC. (n = 30) 0 (0.0%) 0 (0.0%) 6 (20.0%) | 24 (80.0%) 0 (0.0%) 
LDC provided adequate technical 

support for issues with the CoreTools 0 (0.0%) 0 (0.0%) 14 (46.7%) | 14 (46.7%) | 2 (6.7%) 
online platform. (n = 30) 

LDC offered sufficient professional 

development opportunities for me to 

ee ieeRack 0(0.0%) | 1(3.3%) | 12 (40.0%) | 16 (53.3%) | 1 (3.3%) 
(n = 30) 

LDC coaches were able to connect me 

with additional resources when 0 (0.0%) 0 (0.0%) 11 (36.7%) | 19 (63.3%) | 0 (0.0%) 
needed. (n = 30) 

It was challenging to coordinate with 

our LDC coach on how to structure " r r : 
Hhofesciotaliiednilie Commibintey 8 (26.7%) 8 (26.7%) 4 (13.3%) 9 (30.0%) 1 (3.3%) 
time. (n = 30) 

When | reached out to our LDC coach, 

he arshe respanded quickly (ae a0 1 (3.3%) 0 (0.0%) 7 (23.3%) | 22 (73.3%) 0 (0.0%) 
a an coach was easy towork with. | 4 (3 30) | (0.0%) | 6 (20.0%) | 23(76.7%) | 0 (0.0%) 
Our LDC coach was knowledgeable 

and provided high quality guidance. 1 (3.3%) 0 (0.0%) 9 (30.0%) | 20(66.7%) | 0 (0.0%) 
(n = 30) 
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Module Creation 


TL13. During the current school year (2018-19), how many LDC modules did your PLC 
individually or collaboratively adapt from existing modules (e.g., modules created in a 
prior year and/or modules from the LDC Library in CoreTools)? 


| | Adapted modules n = 30, Mean = 4.10, Range: 0 - 15 


TL14. During the current school year (2018-19), how many LDC modules did your PLC create 
(either individually or in a group)? Only include modules built from scratch, not those 
adapted from existing modules in the LDC library. 


| | New modules n = 30, Mean = 2.50, Range: 0 - 10 


TL15. How did members of your PLC collaborate to create LDC modules? Check all that 


apply. 
(n = 30) 
Modules were created by individual teachers. 14 Teacher Leaders (46.7%) 
Modules were created by teams of two or more teachers. 20 Teacher Leaders (66.7%) 
Modules were created by the PLC as a whole. 8 Teacher Leaders (26.7%) 
Other (please specify) 2 Teacher Leaders (6.7%) 
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TL16. Please indicate to what extent you were able to do each of the following when 
creating LDC modules. 


Toa 
Not at all A little bit moderate 
extent 


To a great 
extent 


Select focus standards for a writing 


ets 0 (0.0%) 0(0.0%) | 12(40.0%) | 18 (60.0%) 


Create a standards-driven writing 


pein ne 0 (0.0%) 0(0.0%) | 13 (43.3%) | 17 (56.7%) 


Select high quality, complex texts and other 
materials to engage students in deeper 0 (0.0%) 1 (3.3%) 14 (46.7%) | 15 (50.0%) 
learning (n = 30) 


Identify the skills students need to develop 
to complete a writing assignment (n = 30) 0 (0.0%) 0 (0.0%) 16 (53.3%) | 14 (46.7%) 


Create daily lessons to teach the skills a 
student needs to complete a writing 0 (0.0%) 4 (13.3%) | 12 (40.0%) | 14 (46.7%) 
assignment (n = 30) 


Differentiate instruction by incorporating 
multiple ways of thinking, various levels of 0 (0.0%) 5 (16.7%) 15 (50.0%) | 10 (33.3%) 
complexity, and multiple modalities. (n = 30) 


Plan for a variety of methods to assess 
student progress (e.g., rubrics and/or mini- 0 (0.0%) 1 (3.3%) 16 (53.3%) | 13 (43.3%) 
task scoring guides) (n = 30) 


Assess the quality of writing assignments 
and/or instructional plans using Peer 
Review/Curriculum Alignment Rubric (e.g. 0 (0.0%) 1 (3.3%) 16 (53.3%) | 13 (43.3%) 
Task Pitfalls Checklist, rubric indicators) 
(n = 30) 


Make a writing assignment relevant and 


ee ators eee 0 (0.0%) 1 (3.3%) | 15(50.0%) | 14 (46.7%) 
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Impact on Student Learning 


TL17. Please indicate to what extent LDC had a positive effect on students in the following areas. 


Not at all A little Moderately | A great deal 
Reading skills (n = 30) 0 (0.0%) 1 (3.3%) 18 (60.0%) | 11 (36.7%) 
Content knowledge (n = 30) 0 (0.0%) 0 (0.0%) 18 (60.0%) | 12 (40.0%) 
Ability to complete writing assignments (n = 30) 0 (0.0%) 0 (0.0%) 14 (46.7%) | 16 (53.3%) 
Quality of students’ writing (n = 30) 0 (0.0%) 0 (0.0%) 16 (53.3%) | 14 (46.7%) 
College and career ready skills (n = 30) 0 (0.0%) 1 (3.3%) 17 (56.7%) | 12 (40.0%) 


Capacity to analyze and understand the components 


fe aes 0 (0.0%) 0(0.0%) | 16(53.3%) | 14 (46.7%) 


Speaking and listening skills (n = 30) 1 (3.3%) 1 (3.3%) 15 (50.0%) | 13 (43.3%) 


Overall literacy performance (n = 30) 0 (0.0%) 0 (0.0%) 19 (63.3%) | 11(36.7%) 


Performance on assessments throughout the school 


cao 0(0.0%) | 0(0.0%) | 18 (60.0%) | 12 (40.0%) 
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School Administrator Support 


Less than one quarter of LDC PLCs 


About one quarter of LDC PLCs 


About one half of LDC PLCs 


About three quarters of LDC PLCs 


More than three quarters of LDC PLCs 


(n = 29) 


TL18. What proportion of PLC meetings focused on LDC did your school administrator attend? 


10 Teacher Leaders (34.5%) 


9 Teacher Leaders (31.0%) 
4 Teacher Leaders (13.8%) 
3 Teacher Leaders (10.3%) 
3 Teacher Leaders (10.3%) 


The following questions refer to the school administrator who oversees the LDC project at your school. 


TL19. Please indicate the degree to which you agree or disagree with the following statements. 


My school administrator... Strongly Disagree Agree Strongly 
disagree agree 
had a firm understanding of LDC. (n = 29) 4 (13.8%) 2 (6.9%) 16 (55.2%) 7 (24.1%) 
allocated resources such as teacher time, payment, 
administrator time, support staff, sub coverage, etc., 3 (10.3%) 1 (3.4%) 14 (48.3%) | 11 (37.9%) 
to ensure the LDC team could meet. (n = 29) 
encouraged teachers to participate in LDC. (n = 29) 2 (6.9%) 1 (3.4%) 13 (44.8%) | 13 (44.8%) 
expressed concerns that implementing LDC is taking A ‘ 3 és 
time away from other instructional priorities. (n = 29) 7 (24.1%) 7 (24.1%) 9 (31.0%) 6 (20.7%) 
communicated how using LDC’s tools supported i i ‘ 
specific school initiatives and/or goals. (n = 29) 2 (6.9%) 2 (6.9%) 17 (58.6%) 8 (27.6%) 
provided me with feedback about my LDC planning ‘ . ‘ ‘ 
anid/orinsthuction, naa) 5 (17.2%) 2 (6.9%) 19 (65.5%) 3 (10.3%) 
ie i assessment a priority at my school. 3 (10.3%) 1 (3.4%) 17 (58.6%) 8 (27.6%) 
used LDC to implement standards-driven assignments 
2 (6.9%) 2 (6.9%) 16 (55.2%) 9 (31.0%) 


within existing curriculum. (n = 29) 
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Teacher Leader Leadership Role 


TL20. Please indicate the degree to which you agree or disagree with the following statements 
about your role in leading your school’s LDC implementation. 


stronely Disagree Agree Strongly 
disagree agree 


| met regularly with my school administrator to make 


planning decisions around LDC. (n = 29) 3 (10.3%) 3 (10.3%) 17 (58.6%) 6 (20.7%) 


| was involved in discussions about differentiating LDC 
implementation to meet teacher learning needs. 1 (3.4%) 1 (3.4%) 18 (62.1%) | 9 (31.0%) 
(n = 29) 


| was involved in discussions about how to expand LDC 


fe} fo} fo} ie) 
implementation at my school in future years. (n = 29) 1 (3.4%) 4 (13.8%) 16 (55.2%) 8 (27.6%) 


My role as an LDC Teacher Leader allowed me to 
effectively advocate for additional resources on my 1 (3.4%) 4 (13.8%) | 17(58.6%) | 7 (24.1%) 
campus. (n = 29) 


| was involved in adjusting the problems of practice 


that my school targeted with the LDC work. (n = 29) 1 (3.4%) 3 (10.3%) 17 (58.6%) 8 (27.6%) 


| met regularly with my LDC coach to manage the LDC 


work pensiaeee 0(0.0%) | 1(3.4%) | 17(58.6%) | 11 (37.9%) 


| feel that my position as an LDC Teacher Leader 
allowed me to build my capacity as an instructional 1 (3.4%) 2 (6.9%) 15 (51.7%) | 11 (37.9%) 
leader among my colleagues. (n = 29) 


| am confident that | can lead our LDC PLC in the 
future without the assistance of an LDC coach. 1 (3.4%) 5 (17.2%) | 14(48.3%) | 9 (31.0%) 
(n = 29) 
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Alignment 


TL21. Please indicate the degree to which you agree or disagree with the following statements. 


Strongly Disagree Agree suonely 
disagree agree 


Our school connected LDC implementation to our 


specific schoolwide goals. (n = 29) 2 (6.9%) 2 (6.9%) 17 (58.6%) 8 (27.6%) 


LDC helped teachers create writing assignments to 


use within their current curricula. (n = 29) 1 (3.4%) 4 (13.8%) 15 (51.7%) 9 (31.0%) 


LDC complemented other initiatives taking place in 


sapechoclameis 2 (6.9%) | 1(3.4%) | 19(65.5%) | 7 (24.1%) 


| view LDC as a strategy for implementing my state’s 


10) i) fe} 9 
College- and Career-Ready Standards. (n = 29) 1 (3.4%) 1 (3.4%) 17 (58.6%) 10 (34.5%) 


The time spent implementing LDC interfered with 


other important initiatives at my school. (n = 29) 3 (10.3%) | 14 (48.3%) 8 (27.6%) 4 (13.8%) 


LDC helped prepare students in my school for 


current state assessments. (n = 29) 1 (3.4%) 1 (3.4%) 20 (69.07) 728.07) 


It was difficult for teachers to focus on LDC because 


of other competing priorities at the school. (n = 29) 1m) 10 (34.5%) | 10(34.5%) 827-670) 


Our instructional leaders are using LDC to implement 
standards-driven assignments within the existing 2 (6.9%) 4 (13.8%) 16 (55.2%) 7 (24.1%) 
curriculum. (n = 29) 
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410) Scale-up and Sustainability 


TL22. Please indicate the degree to which you agree or disagree with the following statements. 


strongly Disagree Agree Stronely 
disagree agree 

| expect that most teachers participating in 

LDC this year will continue to do so next 2 (6.9%) 6 (20.7%) 14 (48.3%) 7 (24.1%) 


year. (n = 29) 


Teachers at my school who were not part of 
the LDC PLC meetings used the LDC 
planning process and/or LDC CoreTools. 

(n = 29) 


As a result of LDC, new collaborations 
across grades and/or subjects were created 3 (10.3%) 6 (20.7%) 13 (44.8%) 7 (24.1%) 
or are being launched at my school. (n = 29) 


5 (17.2%) | 11(37.9%) | 7 (24.1%) 6 (20.7%) 


Teachers and administrators at my school 
are committed to sustaining the LDC 3 (10.3%) 5 (17.2%) 15 (51.7%) 6 (20.7%) 
initiative. (n = 29) 


| expect our LDC PLC to increase in size next 


jean 3 (10.3%) | 8(27.6%) | 12 (41.4%) 6 (20.7%) 


10) District Support 


TL23. Please indicate the degree to which you agree or disagree with the following statements. 


Strongly pleaeree eres Strongly Don’t know 
disagree agree 


District leaders supported the 


peace eae 1 (3.4%) | 1(3.4%) | 8(27.6%) | 2(6.9%) | 17 (58.6%) 


District leaders had a firm understanding of 


we aw 1 (3.4%) | 1(3.4%) | 7(24.1%) | 1(3.4%) | 19 (65.5%) 


District leaders are interested in spreading 


the use of LDC to additional schools. (n = 29) 1 (3.4%) 0 (0.0%) a(24.1%) 2 (6.9%) 19 (65.5%) 


District professional development efforts 


were aligned with the LDC initiative. (n = 29) 1 (3.4%) 2 (6.9%) 7 (24.1%) 2 (6.9%) 17 (58.6%) 


District leaders visited my school to discuss 


the implementation of LDC. @a26l 2 (6.9%) | 3 (10.3%) 6 (20.7%) 2 (6.9%) 16 (55.2%) 
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ily, Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 


e CoreTools online platform 
e LDC online courses in the “LEARN” section of CoreTools 
e Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e In-person coaching 


o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


TL24. What supports did you find the most useful and why? 


TL25. What supports were not helpful and why? 


TL26. In what ways could LDC implementation be improved in your school in the future? 
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Appendix C: 
Administrator Survey and Responses 


2018-2019 
LDC School Administrator Survey 


LDC Participation 


A1. What is your role at the school? 


(n = 16) 
Principal 8 admins (50.0%) 
Assistant Principal 8 admins (50.0%) 
Other (please specify) 0 admin (0.0%) 


Professional Learning Community 


A2. What proportion of LDC Professional Learning Community (PLC) meetings did you 
attend during the current school year? 


(n= 15) 


Less than one quarter of LDC PLCs 4 admins (26.7%) 


About one quarter of LDC PLCs 3 admins (20.0%) 
About one half of LDC PLCs 3 admins (20.0%) 
About three quarters of LDC PLCs 3 admins (20.0%) 


More than three quarters of LDC PLCs 2 admin (13.3%) 


Training and Support 


A3. How many in-person and/or online LDC professional development offerings for school 
administrators and teacher leaders did you attend during the current school year (e.g., 
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Summer launch meetings, quarterly in-person administrator meetings, monthly virtual 


LDC leadership meetings)? 


| | LDC Professional development offerings n= 15, Mean = 6.00, Range: 0 - 20 


A4. Please indicate the degree to which you agree or disagree with the following statements. 


strongly Disagree Agree Strongly N/A 
disagree agree 
| was able to reach LDC staff when | had 0 (0.0%) 0 (0.0%) 6 (40.0%) 9 (60.0%) 0 (0.0%) 
questions about LDC. (n = 15) 
My school has adequate technology to 0 (0.0%) 1 (6.7%) 5 (33.3%) 9 (60.0%) 0 (0.0%) 
access LDC online resources. (n = 15) 
LDC offered sufficient professional 0 (0.0%) 0 (0.0%) 6 (40.0%) 9 (60.0%) 0 (0.0%) 
development opportunities for LDC 
teacher leaders. (n = 15) 
LDC offered sufficient professional 0 (0.0%) 0 (0.0%) 8 (53.3%) 7 (46.7%) 0 (0.0%) 
development opportunities for school 
administrators. (n = 15) 
LDC staff members were able to connect 0 (0.0%) 0 (0.0%) 7 (46.7%) 7 (46.7%) 1 (6.7%) 
me with additional resources when 
needed. (n = 15) 


Classroom Observation 


A5. On average, how many times during the school year did you observe each member of 


the LDC PLC teaching an LDC module? 


0 times y Skip to A7 


1 time 


2 times 


3 or more times 


(n = 15) 

0 admin (0.0%) 

2 admins (13.3%) 
4 admins (26.7%) 
9 admins (60.0%) 
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A6. On average, how effective were LDC modules in developing students’ literacy skills? 


(n = 15) 
Not effective 0 admin (0.0%) 
A little effective 0 admin (0.0%) 
Moderately effective 9 admins (60.0%) 
Very effective 6 admins (40.0%) 


Impact on Teacher Practice 


A7. Based on your oversight of the LDC program, please indicate on average how much the 
teaching practice of LDC PLC members improved in each of the following areas: 


Not at all A little Moderately | A great deal 
Selecting a set of focus standards for a writing 0 (0.0%) 0 (0.0%) 7 (46.7%) 8 (53.3%) 
assignment (n = 15) 
Creating standards-driven writing assignments (n = 15) 0 (0.0%) 0 (0.0%) 7 (46.7%) 8 (53.3%) 
Identifying the skills students need to develop to 0 (0.0%) 1 (6.7%) 6 (40.0%) 8 (53.3%) 
complete a writing assignment (n = 15) 
Creating daily lessons to teach each skill a student needs 0 (0.0%) 1 (6.7%) 7 (46.7%) 7 (46.7%) 
to complete a writing assignment (n = 15) 
Systematically collecting information on students’ 0 (0.0%) 0 (0.0%) 9 (60.0%) 6 (40.0%) 
progress. (n = 15) 
Identifying patterns of student understandings or 0 (0.0%) 3 (20.0%) 6 (40.0%) 6 (40.0%) 
misconceptions (n = 15) 
Using evidence of student progress on standards to 0 (0.0%) 2 (13.3%) 8 (53.3%) 5 (33.3%) 
modify subsequent instruction (n = 15) 
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Impact on Student Learning 


A8. Please indicate to what extent LDC had a positive effect on students in the following 


areas: 


school year (n = 15) 


Not at all A little Moderately | A great deal 

Reading skills (n = 15) 0 (0.0%) 2 (13.3%) 8 (53.3%) 5 (33.3%) 
re ere eer 0 (0.0%) 3 (20.0%) 7 (46.7%) 5 (33.3%) 

0, 9 0, 0, 
Ability to complete writing assignments (n = 15) OTe) al oe) Pore) 
Quality of students’ writing (n = 15) oe) sah once oe) nee 

0, 0, 0, 0, 
College and career ready skills (n = 15) aE) ie Peon) Se 
Capacity to analyze and understand the 0 (0.0%) 4 (26.7%) 4 (26.7%) 7 (46.7%) 
components of a writing assignment (n = 15) 

0, 9 0, 0, 
Speaking and listening skills (n = 15) oe) sa aati PGT) es 

0, 9 0, 0, 
Overall literacy performance (n = 15) ee sa ai TG) ee 
Performance on assessments throughout the 0 (0.0%) 3 (20.0%) 6 (40.0%) 6 (40.0%) 
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7 


Administrator Leadership Role 


A9. Please indicate the degree to which you agree or disagree with the following 


statements about your role in leading LDC implementation in your school: 


Strongly Disagree Agree seonely 
disagree agree 
| was able to shape LDC implementation at my school. 0 (0.0%) 3 (20.0%) 7 (46.7%) 5 (33.3%) 
(n= 15) 
| met regularly with the LDC teacher leader in my school 0 (0.0%) 2 (13.3%) 5 (33.3%) 8 (53.3%) 
to stay abreast of implementation progress. (n = 15) 
| was involved in discussions about differentiating LDC 0 (0.0%) 1 (6.7%) 7 (46.7%) 7 (46.7%) 
implementation to meet teacher learning needs. (n = 15) 
| led discussions about how to expand my school’s LDC 0 (0.0%) 2 (13.3%) 8 (53.3%) 5 (33.3%) 
implementation in future years. (n = 15) 
| made changes to school schedules to accommodate LDC 0 (0.0%) 2 (13.3%) 5 (33.3%) 8 (53.3%) 
professional learning time. (n = 15) 
| allocated resources such as teacher time, payment, 0 (0.0%) 2 (13.3%) 6 (40.0%) 7 (46.7%) 


administrator time, support staff, sub coverage, etc., to 
ensure the LDC team could meet. (n = 15) 
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Alignment 


A10. Please indicate the degree to which you agree or disagree with the following 


statements: 

stonely Disagree Agree strongly 

disagree agree 
Our school connected LDC implementation to our 0 (0.0%) 2 (13.3%) 8 (53.3%) 5 (33.3%) 
specific schoolwide goals. (n = 15) 
LDC helped teachers create writing assignments to 0 (0.0%) 0 (0.0%) 9 (60.0%) 6 (40.0%) 
use within their current curricula. (n = 15) 
LDC complemented other initiatives taking place in 0 (0.0%) 1 (6.7%) 7 (46.7%) 7 (46.7%) 
my school. (n = 15) 
| view LDC as a strategy for implementing my state’s 0 (0.0%) 1 (6.7%) 9 (60.0%) 5 (33.3%) 
College- and Career-Ready Standards. (n = 15) 
The time spent implementing LDC interfered with 4 (26.7%) 6 (40.0%) 2 (13.3%) 3 (20.0%) 
other important initiatives at my school. (n = 15) 
LDC helped prepare students in my school for current 0 (0.0%) 1 (6.7%) 8 (53.3%) 6 (40.0%) 
state assessments. (n = 15) 
It was difficult for teachers to focus on LDC because 4 (26.7%) 6 (40.0%) 2 (13.3%) 3 (20.0%) 
of other competing priorities at the school. (n = 15) 
| am using LDC to implement standards-driven 0 (0.0%) 1 (6.7%) 8 (53.3%) 6 (40.0%) 
assignments within the existing curriculum. (n = 15) 
LDC helped improve teacher evaluation ratings. 0 (0.0%) 3 (20.0%) 7 (46.7%) 5 (33.3%) 
(n = 15) 
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Scale-up and Sustainability 


A11. Please indicate the degree to which you agree or disagree with the following 


statements: 
Strongly eee poee Strongly Don’t 
disagree agree know 
| expect that most teachers participating in LDC this 0 1 5 5 3 
year will continue to do so next year. (n = 14) (0.0%) (7.1%) (35.7%) (35.7%) (21.4%) 
Teachers at my school who were not part of the LDC 
PLC meetings used the LDC planning process and/or . 6 2 2 4 
9 0, 9 9 9 
LDC CoreTools. (n = 14) (7.1%) (42.9%) (21.4%) (14.3%) (14.3%) 
As a result of LDC, new collaborations across grades 
and/or subjects were created or are being launched . . . J . 
0, 9 9 9 9 
ati echGal Inet) (0.0%) (14.3%) (57.1%) (21.4%) (7.1%) 
Teachers and administrators at my school are 0 1 7 4 2 
committed to sustaining the LDC initiative. (n = 14) (0.0%) (7.1%) (50.0%) (28.6%) (14.3%) 
| expect our LDC PLC to increase in size 0 2 6 3 3 
next year. (n = 14) (0.0%) (14.3%) (42.9%) (21.4%) (21.4%) 
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110) District Support 


A12. Please indicate the degree to which you agree or disagree with the following 


statements: 
Strongly Sein Ase Strongly Don’t 
disagree agree know 
District leaders supported the 1 (7.1%) 1 (7.1%) 5 (35.7%) 2 (14.3%) 5 (35.7%) 
implementation of LDC. (n = 14) 
District leaders had a firm understanding 1 (7.1%) 1 (7.1%) 6 (42.9%) 0 (0.0%) 6 (42.9%) 


of LDC. (n = 14) 


District leaders are interested in spreading 1 (7.1%) 0 (0.0%) 4 (28.6%) 0 (0.0%) 9 (64.3%) 
the use of LDC to additional schools. 
(n= 14) 


District professional development efforts 1 (7.1%) 1 (7.1%) 4 (28.6%) 3 (21.4%) 5 (35.7%) 
were aligned with the LDC initiative. 
(n = 14) 


District leaders visited my school to discuss 2 (14.3%) 4 (28.6%) 5 (35.7%) 0 (0.0%) 3 (21.4%) 
the implementation of LDC. (n = 14) 
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iM) Areas for Improvement 


There have been a number of supports for implementation of LDC in your school, including: 
e CoreTools online platform 
e LDConline courses in the “LEARN” section of CoreTools 
e Virtual coaching 
o Zoom meetings, written feedback on teacher work in LDC CoreTools, emails, etc. 
e  /In-person coaching 


o Summer training, in-person support visits from LDC and District Lead, in-person 
professional development opportunities, etc. 


A13. What supports did you find the most useful and why? 


A14. What supports were not helpful and why? 


A15. In what ways could LDC implementation be improved in your school in the future? 
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Appendix D: 
LDC Module Rating Dimensions 


Each module was rated on six dimensions. All of these were rated using a 5-point scale. With the first 
five, anchor points ranged from not present or realized to fully present or realized. In contrast, the 
overall dimension ranged from inadequate to advanced. 


Dimension 


1 Effective Writing Task 


Definition 

Degree to which teaching task makes effective use of the template task’s writing mode (i.e., argumentation 
or explanation); requires sustained writing and effective use of ideas and evidence to substantiate claims; 
and is feasible for most students to complete (i.e., appropriate for the grade-level and subject matter). 


Main Sources of Information 


Module Creator Handout (Task) 


e Read and evaluate the teaching task, student background/prior knowledge, and summary information. 
e Evaluate the difficulty or ease students may encounter trying to answer the question. 


e Compare module teaching task to teaching task template options. 


Anchor Points Description 


5 Fully Present or Realized The teaching task and performance expectations for the module are 
explicit and clear, require students to engage in higher-order 
thinking and writing, and are appropriate for the grade-level and 
subject matter. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized | Clear module teaching task and performance expectations are 
available, but do not require students to engage in higher-order 
thinking and writing and/or are not appropriate for the grade-level 
and subject matter. 


2 Barely Present or Realized 


1 Not Present or Realized There is minimal evidence of an effort to identify an explicit and 
clear teaching task and performance expectations that provide 
opportunities for students to think critically and are appropriate for 
the grade-level and subject matter. 
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Dimension 


2 


Alignment to CCSS, Local, State Literacy and Content Standards 


Definition 


Extent to which module addresses content essential to the discipline, as well as reading comprehension and 
writing standards informed by local and state standards. 


Main Sources of Information 


Module Creator Handout (Task) 


e Read and evaluate the standards included in the module. 


e Module should include ELA as well as subject matter CCSS/state standards. 


e Compare and contrast the standards the module includes with those that could have been included. 


e Particular attention to content standards (CCSS History/Social Studies, Science, and Technical Subjects); 


State Standards; Specific Reading, Writing, Speaking/Listening, Language Skills 


Anchor Points 


Description 


5 Fully Present or Realized 


Module specifically addresses content essential to CCSS and local or 
state standards in science or social studies, as well as reading 
comprehension and writing. All standards are well aligned to the 
topic and teaching task. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


Module broadly addresses content essential to CCSS and local or 
state standards in science or social studies and reading 
comprehension and writing. Standards are sufficiently aligned to the 
topic and teaching task. 


2 Barely Present or Realized 


1 Not Present or Realized 


Minimal evidence that module addresses content essential to the 
discipline and literacy standards. Standards are poorly aligned to the 
topic and teaching task. 
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Dimension 


3 


Fidelity to LDC Module Instruction 


Definition 


Degree to which module instruction, activities, and teaching task address each of the four stages of 


instructional practice (preparation for the task, reading process, transition to writing, writing process). 


Main Sources of Information 


Module Creator Handout (Instruction) and Information Sheet 


e Evaluate for distribution of activities and time spent on each of the four stages of instructional practice. 


Anchor Points 


Description 


5 Fully Present or Realized 


The module instruction, activities, and teaching task reflect 
deliberate attention and fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect demonstrable effort 
to develop instructional scaffolding within and across each stage of 
instruction. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


The module instruction, activities, and teaching task reflect 
moderate attention and fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect sufficient effort to 
develop instructional scaffolding within and across each stage of 
instruction. 


2 Barely Present or Realized 


1 Not Present or Realized 


The module instruction, activities, and teaching task reflect poor 
attention and lack of fidelity to the four discrete stages of LDC 
module instruction. Classroom materials reflect inadequate effort to 
develop instructional scaffolding within and across each stage of 
instruction. 
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Dimension 


A 


Quality Instructional Strategies 


Definition 


Degree to which the module provides clear instructional strategies aimed at helping students develop literacy 


skills and successfully complete the teaching task. In addition, the degree to which module instruction and 


activities scaffold critical thinking and performance in a way that is meaningful within the context of a given 


field or subject matter. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


e Evaluate extent to which the module activities scaffold critical thinking and student performance within 


the context of the subject matter at the core of the teaching task. 


e Evaluate extent to which instructional strategies guide student learning in literacy and ability to complete 


the teaching task. 


Anchor Points 


Description 


5 Fully Present or Realized 


Module provides clear and targeted instructional strategies and 
activities that scaffold student learning and promote critical thinking 
in social studies or science. There is explicit attention to helping 
students develop an accurate understanding of the topic and 
teaching task, and literacy skills necessary to complete the writing 
task successfully. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


Instructional strategies and activities are available to support 
adequate student learning and critical thinking in social studies or 
science. There is moderate attention to helping students develop an 
understanding of the topic and teaching task, and literacy skills 
necessary to complete the writing task. 


2 Barely Present or Realized 


1 Not Present or Realized 


Limited instructional strategies and activities are available to support 
student learning and critical thinking in social studies or science. 
Insufficient attention to helping students develop an understanding 
of the topic and teaching task, or literacy skills necessary to 
complete the writing task. 


188 


Dimension 


ts) 


Coherence and Clarity of Module 


Definition 


The degree of logical alignment found between the teaching task and the goals of the module with the 


readings, mini-tasks, and instructional strategies. 


Main Sources of Information 


Module Creator Handout (Instruction), Classroom Handouts, and Student Work 


Anchor Points 


Description 


5 Fully Present or Realized 


Strong alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 


4 Sufficiently Present or Realized 


3 Moderately Present or Realized 


Moderate alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 


2 Barely Present or Realized 


1 Not Present or Realized 


Poor alignment between the teaching task and goals of the 
module—including the CCSS and local and state literacy and content 
standards—with the readings, mini-tasks, student work, and 
instructional strategies. 
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Dimension 


6 Overall Impression 


Definition 
A holistic assessment of the LDC Module. 


Main Sources of Information 


Module Creator Handout, Classroom Handouts, and Student Work 


e To what extent does this module contribute to student college readiness and development of advanced 
literacy skills? 


Anchor Points 


5 Advanced LDC Module Implementation 


Proficient LDC Module Implementation 


4 
3 Adequate LDC Module Implementation 
2 


Marginal LDC Module Implementation 


- 


Inadequate LDC Module Implementation 
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Appendix E: 
Principal Interviews 


To better understand teacher retention rates of LDC schools after the 2016-2017 school 
year, we conducted an interview study of school administrators. Twenty principals and assistant 
principals volunteered to be interviewed. This section presents the methods and results of the 
principal interviews. 


Method 
Instrument 


We developed a six-question interview protocol for principals and assistant principals. 
Some of the interview questions had sub-questions. Five of the questions were asked of all 
principals, regardless of their schools’ retention typology, and one question (question 5) 
differed slightly based on the school’s retention typology. See Figure E1 for the interview 
protocol. 


Figure E1 
Principal and assistant principal interview protocol. 
Question 
No. Question Given to 
1 Why did you initially participate in LDC? All 


e What were your goals when you signed on and how well have those 
goals been met thus far? 
2 What was your role in LDC implementation? All 
e Who else at the school supported LDC? 
e How much did you see of PLC meetings? 
e Was part of your role to give feedback on LDC instruction? (If 
needed: Who gave feedback on LDC instruction?) 


3 Could you tell me about outside support — from both LDC and the district — All 
and what worked or didn’t work? 
4 What were the incentives for teachers’ participation in LDC? All 


e How would you describe teacher buy-in and commitment to LDC? 
e Did teachers receive any additional pay? If so, for what? 


5 What would you say are the main factors that led to not continuing with Drop out 
LDC? 
a) Looking across the district, we noticed that some schools chose to Zero 
continue with LDC for another year and others chose to stop. What retention 


would you say are the main factors that led to continuing with LDC? 

b) We noticed that none of the year-1 teachers participated in year 2, so 
the year-2 PLC began with an entirely new set of teachers. Why do you 
think that was? 

c) Were there changes in implementation to keep this from happening 
again the following year? 


191 


a) Looking across the district, we noticed that some schools chose to Very Low 
continue with LDC for another year and others chose to stop. What and Low 
would you say are the main factors that led to continuing with LDC? Retention 

b) We noticed that many teachers who participated in Year 1 did not 
participate in Year 2. Why do you think that was? 


a) Looking across the district, we noticed that some schools chose to Moderate- 
continue with LDC for another year and others chose to stop. What High 
would you say are the main factors that led to continuing with LDC? Retention 


b) We noticed that many teachers who participated in Year 1 continued 
into Year 2. Why do you think that was? 


6 Do you have any advice for strengthening LDC or is there anything else we All 
should know about your experience with LDC? 
Participants 


During the 2016-2017 school year, 48 total schools participated in the i3 LDC program 
including 28 in the New York City Department of Education (NYCDOE) and 20 in the West Coast 
district which is the subject of this report. We categorized the schools into retention typologies 
based on a school’s total teacher participants in 2016-2017 and returning teachers in 2017- 
2018. Using the proportions of the same teacher participating for the 2 years, we created five 
school retention typologies: 


drop out (the school has no teachers participating in LDC in 2017-2018); 


zero retention (no 2016-2017 teachers returning to LDC, but new teachers in the 
school are participating in 2017-2018); 


very low retention (less than 1/3 of teachers returning); 
low retention (between 1/3 and 2/3 returning teachers); and 


moderate to high retention (over 2/3 teachers returning). 


The following table show the number of schools per retention typology in NYCDOE and 
the West Coast district. 
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Table E1 


Number of Schools and Proportion of Retention Typologies for NYCDOE and West Coast district in 2016— 


2017 
Number of — Proportion of 
Number of | Proportionof West Coast total West 
NYCDOE total NYCDOE district Coast district 
Retention type schools schools schools schools 
Dropped out after 2016-2017 10 36% 6 30% 
Zero retention 1 4% 2 10% 
Very low teacher retention 7 25% 3 15% 
Low teacher retention 5 18% 2 10% 
Moderate to high teacher retention 5 18% 7 35% 
Total 28 20 


All LDC schools from the 2016-2017 cohort were contacted for a phone interview by 


email, and a total of 20 principals and assistant principals (10, or 36%, of NYCDOE schools and 


10, or 50%, of West Coast district schools) participated in the interview. Table E2 shows the 


school retention typologies that are represented in this sample. Of the principals and assistant 
principals who participated in an interview, most were from very low teacher retention schools 
(8 participants), followed by moderate to high teacher retentions schools (6 participants). We 
also had three principals or assistant principals from drop out schools participate, two principals 
from low teacher retention schools, and one principal in a zero-retention school. 
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Table E2 
School Retention Typologies Represented in the Study 


NumBaRes Proportion 
Number of Proportion Sener of West 
participating of NYCDOE ee oe Coast 
NYCDOE schools in es district 
district . 
schools the typology schools in 
; schools 
Retention Type the typology 
Dropped out after 2016-2017 2 20% 1 17% 
Zero retention 0 0% 1 50% 
Very low teacher retention 5 71% 3 100% 
Low teacher retention 1 20% 1 50% 
Moderate to high teacher retention 2 40% 4 57% 
Total 10 10 


Procedures 


We originally contacted principals of all schools except for schools designated as low 
retention. This was done as sampling strategy for a case study methodology to understand 
reasons behind high and low retention schools. Initial emails were sent to 41 schools in both 
districts (23 in NYCDOE and 18 in the West Coast district) on the week of June 16, 2018. A 
minimum of three follow-up emails for non-responding administrators were sent between June 
25 and August 6, 2018. During this window, nine NYCDOE principals or assistant principals 
participated (39% of schools in the first round) and seven principals from the West Coast 
district participated (39% of schools in the first round). One West Coast district principal 
contacted us to say that she did not have enough knowledge about the LDC program in the 
school because she was the new principal. 


Because of the low response rate, we decided to contact the principals in the low teacher 
retention schools (five in NYCDOE and two in the West Coast district) to increase the numbers 
of participants. The initial email for these NYCDOE principals were sent on September 20, 2018 
and, for West Coast district principals, between October 1 and October 8, 2018. Follow-up 
emails were sent to NYCDOE and West Coast district principals between the weeks of October 1 
and October 19, 2018. We also reached out to the LDC grant directors in NYCDOE and the West 
Coast district for recruitment support. Between the new round of emails and support from the 
LDC grant directors, four additional school leaders participated in interviews, bringing our total 
participants to 20 principals and assistant principals (10, or 36%, of NYCDOE schools and 10, or 
50%, of West Coast district schools). 


Interviews were scheduled at times most convenient to the principal or assistant 
principal. Interviews were conducted by two researchers following the protocol, and the 
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interviews ranged from about eight minutes to 30 minutes in length. Participant consent was 
obtained prior to the interviews, and interviews were recorded. 


Analysis 


Recordings of interviews were transcribed. (For one interview, due to researcher and 
equipment error, the interview was not recorded. However, notes taken by the researcher 
during the interview were used for general counts in the analysis.) Transcriptions were read 
iteratively and coded for broad themes using grounded theory (Glaser & Strauss, 1967). 
Multiple close readings of each group transcript were conducted in order to summarize the 
data and identify themes related to reasons for a school’s retention (or lack thereof) of teachers 
from year 1 to year 2 of LDC implementation. A coding scheme was developed based on 
targeted areas of interest in relation to reasons for retention. 


Results 


Qualitative results from the data are presented in five main categories: reasons for 
teachers continuing in LDC, meeting goals, administrator’s role, teacher buy-in, and outside 
support. 


Reasons for Teachers Continuing or Leaving LDC in Year 2 


To understand why teachers remained or left LDC after year 1, interview data were coded 
into three main categories: teacher or grade level decision, administrative decision, and 
mandatory participation. In teacher or grade level decisions, individual teachers, a grade level, 
or a content-area team (e.g., English teachers) were free to decide to return the following year. 
Administrative decisions typically involved the principal deciding on changing the grade level or 
content-area teachers who would participate in the second year of implementation (e.g., 
moving from second grade to third grade teachers; dropping mathematics teachers but keeping 
science teachers in the program). A couple principals also decided that LDC participation would 
be better suited for a particular level of teacher. For example, a principal decided to focus year 
2 teacher participation on “teachers that were new to the profession and teachers that were 
struggling instructionally;” therefore, only a few teachers continued from year 1 to year 2 at 
that site. Mandatory participation, which is based on an administrator’s decision, reflected 
schools where teachers were told that participation in the second year of LDC was not 
voluntary. Table E3 shows the reasons for retention by school retention typology. 
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Table E3 
Teachers’ Participation in LDC in Year 2 by Retention Typology 


: Teacher/grade Administrative | Mandatory 
Retention typology 


level decision decision participation 
Dropped out after 2016-2017 1 2 0 
Zero retention 0 1 0 
Very low teacher retention 4 4 0 
Low teacher retention 1 0 1 
Moderate to high teacher retention 4 0 2 
Total 10 7 3 


For the three drop out schools in our sample, two of the schools had a change in 
leadership. One had a new principal who came in before the second year of LDC 
implementation and decided not to participate in LDC, and the other school had a new 
superintendent who did not continue partnering with LDC. For the last of the three drop out 
schools, the small group of teachers who were in the first year of implementation decided not 
to continue because of “teacher and administrator burn out.” This school had too many 
initiatives that were handled by a small number of administrative staff and the same teachers 
who were also part of the LDC implementation. 


For the one zero retention school in our sample, the principal decided to change the 
teachers for year 2. The grade level that was selected to participate in LDC the first year found 
the program successful, and so for year 2, “other teachers haven't been exposed, and we 
wanted more teachers to have access...to spread the wealth within the school,” so another 
grade level was selected to participate. In one of the low retention schools, the principal said 
that participation in LDC was mandatory, but many of teachers who were part of year 1 
implementation left the school the following year. 


For the 10 schools that were coded as a teacher/grade level decision, half of those 
decisions were based on the grade-level or content-area team (e.g., science teachers) deciding 
to leave LDC because they started a new program or initiative the second year and did not have 
the time to participate in LDC. For the moderate to high teacher retention schools who were 
coded as teacher/grade level decision, the majority of teachers in those schools decided to stay 
in LDC the second year. 


Meeting Goals 


We asked participants their reasons for participating in LDC during the first year. The most 
popular reasons for participating in LDC were its alignment to standards and rigorous tasks. 
Principals and assistant principals also stated that they felt both teachers and students in their 
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schools needed assignments where literacy instruction, the writing tasks, and the writing 
process were a focus. Other responses included wanting better lesson planning, collaboration, 
and content-area instruction. Interestingly, three principals said that LDC appealed to them as 
they were new to the school or position. 


After stating their reasons for participation, we asked if these goals were met after the 
first year of implementation. Table E4 shows participant responses to whether goals were met 
by school typology. The majority of principals and assistant principals responded positively: 12 
(60%) said yes, five expressed mixed feelings about accomplishing their original goals for LDC 
(25%), and three said that their goals were not met (15%). From these responses, meeting goals 
for LDC did not appear to influence retention of teachers. For example, two of the three 
dropout schools reported meeting their LDC goals for participation, yet they didn’t remain in 
the program. Teacher retention seemed to be tied more to teacher and principal decisions (see 
section above). 


Table E4 
Principal Reponses for LDC Goals by School Typology 


LDC goals met 


School typology 


Yes No Mixed 
Dropped out after 2016-2017 2 1 0 
Zero retention 1 0 0 
Very low teacher retention 4 1 3 
Low teacher retention 1 1 0 
Moderate to high teacher retention 4 0 2 
Total 12 3 5 


Administrator’s Role 


The majority of principals and assistant principals in our sample reported having an active 
role in supporting LDC implementation. Sixteen (80%) attended LDC PLCs, and those who could 
not make the PLCs were able to get a summary of the meetings from their assistant principals 
or lead teachers. For those who were able to attend some PLCs, the amount of time they were 
able to sit in on the PLCs varied. Some principals were only able to drop in for a portion of the 
PLCs. For example, one principal (moderate to high retention school) said, “| would probably 
just go in the last 15 minutes. | don't think | was in a full session all the time. But | would touch 
bases and then each teacher would give me a wrap up summary of what they had done, what 
they were working on." Others were able to go to almost all PLCs. As the zero-retention school 
principal said, “| was a participant, also, in the learning.” Sixteen participants (80%) also 
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reported that they gave feedback on LDC instruction. However, the degree to which principals 
or assistant principals gave feedback varied. Table E5 shows the numbers of participants who 
attended PLCs and provided feedback on LDC instruction. 


Table E5 
Participation of Principals and Assistant Principals in LDC Implementation by Retention Type 


Retention typology Attended PLCs Provided feedback 
Dropped out after 2016-2017 3 3 
Zero retention 1 1 
Very low teacher retention 6 6 
Low teacher retention 1 2 
Moderate to high teacher retention 5 4 
Total 16 16 


Teacher Buy-In 


Most of the principals and assistant principals mentioned increasing skills and knowledge, 
collaboration, and student achievement and accessing LDC materials as incentives for teachers. 
Besides these incentives that were actually part of the LDC implementation, few schools were 
able to provide additional school-based supports. Eight schools (40%) in our sample reported 
providing teachers with more tangible incentives for their participation in LCD, such as 
additional pay and time, and four of the eight gave multiple incentives. Table E6 shows the 
number of these types of teacher incentives by school typology. No pattern emerges that 
connects teacher incentives to retention typology. However, it is notable that the majority of 
schools in the sample did not provide tangible incentives (i.e., pay, extra time, teacher 
evaluation) to teachers for their participation for LDC. 
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Table E6 
Teacher Incentives by School Typology 


Retention typology Additional pay Extra time Teacher evaluation 


Dropped out after 2016-2017 
Zero retention 

Very low teacher retention 
Low teacher retention 


Moderate to high teacher retention 


ule PN FP Oo 
BlP N O F O 
NirFPF oO F ODO O 


Total 


Five schools (25%) were able to pay teachers for their participation in LCD. Aside from 
pay, some principals were able to support participating teachers with extra time, such as 
adjusting the schedule to give teachers more time to meet during the week or giving teachers a 
release day. For example, the principal from the zero-retention school said, “Initially, it was kind 
of an imposition of time, but when we created extra time and space for the teachers to meet, 
they were appreciative. So, they had more buy-in.” Another principal (low retention school) 
said, “I make it very easy for teachers, because I've built in time in their schedule for them to 
meet for LDC. There are no discrepancies in terms of when we should meet or how can we 
meet, or what needs to be done, which is usually the issue sometimes when it comes to work 
with outside consultants or providers, right?” 


Two schools incentivized participation through teacher evaluations. One principal (very 
low retention school) postponed teacher evaluations for the year. The other principal 
(moderate to high retention school) said, “Il very much tied it to teacher performance and my 
expectation of what high quality planning looks like.” For this principal, he knew LDC would be 
“transformative. | knew that if they got through it and actually did it, the carrot would reveal 
itself in the students’ performance. And | knew my teachers were committed; | knew they 
wanted our kids to do well. They just needed to be pushed through the process to see how to 
get the kids to do wel 


| 


Outside Support 


Overwhelmingly positive responses were reported from our sample regarding LDC 
support the schools received. All principals and assistant principals mentioned either training or 
coaching that LDC provided. Two principals, both from a very low retention schools, also 
mentioned that the administrator meetings as helpful. As one principal described, “It gave an 
opportunity to speak to other people who were in the project, and it also gave an opportunity 
to actually go through all of the CoreTools that were there to support the teachers.” Two 
principals (both from New York) also found site visits to another school as helpful: “There are 
sites where the work is going on really nicely, and we were able to go over to those sites and 
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see presentations from the principal and her teachers on how they have embedded this entire 
system into their daily program.” Five participants mentioned district support being available. 


Discussion 


Participants in this study all reported playing an active role in LDC implementation at their 
school site, and most said that their goals for participation were met. They also overwhelmingly 
responded favorably to LDC support, and district support, if provided or perceived, was also 
helpful. The principal interviews showed the diversity of reasons for why schools were able to 
retain teachers, and they also show why teachers left. The main reason for teacher retention 
was teacher and grade-level team decisions to leave or stay, followed by the principals’ decision 
to switch participation of teachers from year 1 to year 2. Teacher buy-in also seemed to affect 
teachers’ decisions for remaining or continuing. We did not find any consistent trends for 
retention typology; that is, we cannot characterize reasons for each retention typology and say, 
for example, all drop out schools had these characteristics or most moderate to high retention 
schools had other characteristics. Teacher retention in LDC from year 1 to year 2 was due to a 
myriad of factors that worked in combination at each school. 
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Appendix F: 
Module Artifact Ratings 


The following presents further details of the module analyses. First, we present the 
methodology and results of the generalizability theory studies. This is followed by the 
additional tables for the descriptive analyses. 


Generalizability Study 


Generalizability theory is a statistical framework for examining multiple sources of 
potential error during the scoring process. For each grade band, we first modeled score 
variability across all six dimensions using a two-faceted design, whereby we estimated variance 
components for module by rater by dimension (t*r*d). The goal here was to separate true 
variation in the modules from other potential sources of measurement error. The main effects 
reflect true variation across modules (o2t) and error variance across raters (o2r) and 
dimensions (o2d), while the error term (o2trd,e) reflects unexplained residual error in the 
model. To disentangle the sources of potential error further, we also used a single-faceted 
design to examine potential error within the scoring of each dimension. As with the first set of 
models, the main effect reflects true variation across teachers (o2t) and error variance across 
raters (o2r). 


Elementary Module Results 


Generalizability theory models were conducted to examine potential error in the scoring 
process for the elementary modules. Results from the two-faceted and one-faceted models 
that examine error across and within dimensions are presented in Tables F1 and F2. As would 
be the goal of any rating session, most of the variation found for the elementary modules was 
due directly to differences in the modules (62%) or to differences in the modules by dimension 
(29%). Despite this, 6% of the variation was due either directly or through interaction with the 
raters. Finally, approximately 1% of the variation was unexplained by the two-faceted model 
used. 


As previously mentioned, we also used a one-faceted design to disentangle variation in 
the ratings that was due either directly or through interaction with the dimensions (see 
Table F2). As would be desired, almost all of the variation found for each dimension was due 
directly to differences in the modules (84% to 100.0%). For most of the dimensions, the 
greatest amount of other variation was found due to an interaction between raters and 
modules. For example, for Dimension 1, which measures the effective writing task, 5% of the 
variation was due directly to the raters and 11% was due to an interaction between the raters 
and modules. The exception involved Dimension 3, which measures fidelity to LDC instruction, 
with 9% of the variation due directly to the raters and only 6% being due to the interaction 
between raters and modules. Finally, there was no unexplained error variance detected by the 
models. 
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Table F1 


Generalizability Study of the Elementary Module Ratings across Dimensions (n = 90) 


Source Var. % 
Module (o2t) 0.78 62.23 
Rater (o2r) 0.00 0.00 
Dimension (o2d) 0.02 1.67 
Module* Dimension (o2td) 0.36 28.89 
Rater* Dimension (o2rd) 0.06 4.57 
Module*Rater (o2tr) 0.02 1.21 
Error (o2trd,e) 0.02 1.42 


Table F2 


Generalizability Study of the Elementary Module Ratings for Each Dimension (n = 139) 


Module Rater Module*Rater Error 
(o2t) (o2r) (o2tr) (o2trd,e) 

Dimension Var. % Var. % Var. % Var. % 
1. Effective writing task 1.15 84.02 0.07 4.99 0.15 10.98 0.00 0.00 
2. Standards alignment 0.98 90.56 0.01 0.92 0.09 8.52 0.00 0.00 
3. Fidelity to LDC instruction 1.16 85.53 0.12 8.90 0.08 5.57 0.00 0.00 
4. Quality instructional 1.23 96.31 0.02 1.49 0.03 2.20 0.00 0.00 
strategies 
5. Coherence/clarity of 1.27. 100.00 0.00 0.00 0.00 0.00 0.00 0.00 
module 
6. Overall impression 0.83 97.16 0.00 0.00 0.02 2.84 0.00 0.00 


Note. Negative estimates of variance were changed to zero in order to calculate percentages (see Shavelson & Webb, 1991) 


Secondary Module Results 


Generalizability theory models were conducted to examine potential error in the scoring 
process for the secondary modules. Tables F3 and F4 present results from the two-faceted and 
one-faceted models that examine error across and within dimensions. As would be the goal of 
any rating session, most of the variation found in the ratings was due directly to differences in 
the modules (36%) or to differences in the modules by dimension (60%). Furthermore, only 3% 
of the variation for the secondary modules was due either directly or indirectly to the raters, 
and only 1% of the variation in ratings for the model was unexplained. 
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Table F3 


Generalizability Study of the Secondary Module Ratings across Dimensions (n = 66) 


Source Var. % 
Module (o2t) 0.41 36.05 
Rater (o2r) 0.00 0.00 
Dimension (o2d) 0.00 0.00 
Module* Dimension (o2td) 0.68 59.91 
Rater* Dimension (o2rd) 0.03 2.72 
Module*Rater (o2tr) 0.00 0.32 
Error (o2trd,e) 0.01 1.00 


Table F4 


Generalizability Study of the Secondary Module Ratings for Each Dimension (n = 66) 


Module Rater Module*Rater Error 
(o2t) (o2r) (o2tr) (o2trd,e) 

Dimension Var. % Var. % Var. % Var. % 
1. Effective writing task 1.34 89.72 0.09 5.90 0.01 0.49 0.06 3.89 
2. Standards alignment 0.84 92.90 0.00 0.00 0.03 2.93 0.04 4.17 
3. Fidelity to LDC instruction 1.57 98.96 0.00 0.00 0.00 0.00 0.02 1.04 
4. Quality instructional 1.25 91.77 0.08 5.95 0.03 2.28 0.00 0.00 
strategies 
5. Coherence/clarity of 0.95 87.46 0.02 1.37 0.12 11.17 0.00 0.00 
module 
6. Overall impression 0.54 88.20 0.06 9.90 0.00 0.00 0.01 1.89 


Note. Negative estimates of variance were changed to zero in order to calculate percentages (see Shavelson & Webb, 1991) 


We also used a one-faceted design to disentangle the variance that was due either 


directly or through interaction with the dimensions (see Table F4). As would be desired, 87% to 
99% of the variance found for each dimension was attributable to differences in the modules. 


Only Dimension 6, which measures the raters’ overall impression of module quality, had 


moderate amounts of variation due directly to the raters (10%). Furthermore, Dimension 5, 
which measures coherence and clarity of the module, had 11% of variation due to an 
interaction between modules and raters. Finally, some error variance (1% to 4%) was found for 
four of the dimensions. The greatest amount of this unexplained error was found for the first 
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two dimensions, which measure foundational features of the modules including the 
effectiveness of the writing task and standards alignment (4%, respectively). 


Summary 


Generalizability models were fit for the overall samples for the two grade bands. As would 
be hoped for, when examining the one-faceted models, the vast majority of variation for the 
elementary and secondary ratings were due directly to differences in the modules (62%, 36%) 
or to differing quality in the modules across dimensions (29%, 60%). Likewise, results from the 
two-faceted models showed that the vast majority of variation within dimension was due to 
differences in both the elementary modules (84% to 100%) and secondary modules (87% to 
99%). Despite this, for the elementary modules 11% of the variation for Dimension 1, which 
measures the effectiveness of the writing task, and 9% of the variation for Dimension 2, which 
measures standards alignment, was due to an interaction between the raters and modules. In 
addition, 9% of the variation for Dimension 3, which measures fidelity to LDC instruction, was 
due directly to differences in the elementary raters. The only dimensions for the secondary 
modules that showed moderate amounts of other variation were Dimension 5, with 11% due to 
an interaction between raters and modules, and Dimension 6, with 10% due to differences in 
raters alone. Finally, some unexplained error variance was found for the two-faceted model 
used for the elementary modules and secondary modules (1%, respectively), and some of the 
one-faceted models used for the secondary modules (0% to 4%). 


Descriptive Results 


The following section presents percentages for the elementary and secondary modules. 
This includes the overall results, and then results broken down by content area and then cohort 


group. 
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Table F5 


Distribution (Percentage) of Ratings for the Modules 


Dimension 1 2 3 4 5 
ELEMENTARY (n = 48) 
1. Effective writing task 8.3 16.7 33.3 27.1 12.5 
2. Standards alignment 12.5 18.8 43.8 18.8 6.3 
3. Fidelity to LDC instruction 8.3 18.8 22.9 35.4 14.6 
4. Quality instructional strategies 8.3 22.9 29.2 29.2 10.4 
5. Coherence/clarity of module 8.3 20.8 31.3 29.2 10.4 
6. Overall impression 4.2 22.9 33.3 37.5 2.1 
SECONDARY (n = 61) 
1. Effective writing task 3.3 14.8 36.1 29.5 9.8 
2. Standards alignment 4.9 14.8 41.0 31.1 8.2 
3. Fidelity to LDC instruction 8.2 16.4 34.4 31.1 6.6 
4. Quality instructional strategies 3.3 13.1 21.3 47.5 11.5 
5. Coherence/clarity of module 6.6 14.8 39.3 29.5 9.8 
6. Overall impression 0.0 14.8 45.9 36.1 3.3 
OVERALL (n = 109) 
1. Effective writing task 5.5 15.6 34.9 28.4 11.0 
2. Standards alignment 8.3 16.5 42.2 25.7 7.3 
3. Fidelity to LDC instruction 8.3 17.4 29.4 33.0 10.1 
4. Quality instructional strategies 5.5 17.4 24.8 39.4 11.0 
5. Coherence/clarity of module 7.3 17.4 35.8 29.4 10.1 
6. Overall impression 1.8 18.3 40.4 36.7 2.8 
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Table F6 


Distribution (Percentage) of Ratings for the Elementary Modules by Content Area 


Dimension 1 2 3 4 5 
ELA (n = 24) 
1. Effective writing task 12.5 25.0 33.3 12.5 12.5 
2. Standards alignment 4.2 20.8 50.0 20.8 4.2 
3. Fidelity to LDC instruction 8.3 29.2 25.0 29.2 8.3 
4. Quality instructional strategies 12.5 20.8 25.0 29.2 12.5 
5. Coherence/clarity of module 12.5 20.8 29.2 29.2 8.3 
6. Overall impression 4.2 25.0 33.3 33.3 4.2 
SCIENCE (n = 11) 
1. Effective writing task 0.0 0.0 36.4 54.5 9.1 
2. Standards alignment 27.3 9.1 45.5 18.2 0.0 
3. Fidelity to LDC instruction 0.0 0.0 36.4 27.3 36.4 
4. Quality instructional strategies 0.0 27.3 36.4 27.3 9.1 
5. Coherence/clarity of module 0.0 18.2 45.5 18.2 18.2 
6. Overall impression 0.0 18.2 36.4 45.5 0.0 
SOCIAL STUDIES (n = 13) 
1. Effective writing task 7.7 15.4 30.8 30.8 15.4 
2. Standards alignment 15.4 23.1 30.8 15.4 15.4 
3. Fidelity to LDC instruction 15.4 15.4 7.7 53.8 7.7 
4. Quality instructional strategies 7.7 23.1 30.8 30.8 7.7 
5. Coherence/clarity of module 7.7 23.1 23.1 38.5 7.7 
6. Overall impression 7.7 23.1 30.8 38.5 0.0 
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Table F7 


Distribution (Percentage) of Ratings for the Elementary Modules by Cohort 


Dimension 1 2 3 4 5 

COHORT 2, 2017-2018 (n = 23) 

1. Effective writing task 13.0 13.0 43.5 26.1 4.3 
2. Standards alignment 21.7 17.4 39.1 21.7 0.0 
3. Fidelity to LDC instruction 17.4 21.7 17.4 26.1 17.4 
4. Quality instructional strategies 13.0 34.8 17.4 26.1 8.7 
5. Coherence/clarity of module 13.0 34.8 17.4 21.7 13.0 
6. Overall impression 8.7 39.1 17.4 34.8 0.0 
COHORT 2, 2018-2019 (n = 19) 

1. Effective writing task 5.3 10.5 15.8 36.8 26.3 
2. Standards alignment 0.0 21.1 47.4 15.8 15.8 
3. Fidelity to LDC instruction 0.0 5.3 31.6 47.4 15.8 
4. Quality instructional strategies 5.3 5.3 42.1 36.8 10.5 
5. Coherence/clarity of module 5.3 0.0 47.4 42.1 5.3 
6. Overall impression 0.0 5.3 47.4 42.1 5.3 


Note. Cohort groups with less than five are not presented. 
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Table F8 


Distribution (Percentage) of Ratings for the Secondary Modules by Content Area 


Dimension 1 2 3 4 5 
ELA (n = 31) 
1. Effective writing task 0.0 9.7 38.7 38.7 9.7 
2. Standards alignment 0.0 12.9 41.9 35.5 9.7 
3. Fidelity to LDC instruction 12.9 9.7 25.8 38.7 6.5 
4. Quality instructional strategies 0.0 0.0 16.1 61.3 16.1 
5. Coherence/clarity of module 3.2 0.0 54.8 29.0 12.9 
6. Overall impression 0.0 6.5 38.7 51.6 3.2 
SCIENCE (n = 10) 
1. Effective writing task 0.0 30.0 20.0 20.0 30.0 
2. Standards alignment 20.0 10.0 10.0 40.0 20.0 
3. Fidelity to LDC instruction 0.0 30.0 30.0 20.0 20.0 
4. Quality instructional strategies 10.0 50.0 0.0 30.0 10.0 
5. Coherence/clarity of module 10.0 50.0 0.0 30.0 10.0 
6. Overall impression 0.0 50.0 10.0 30.0 10.0 
SOCIAL STUDIES (n = 20) 
1. Effective writing task 10.0 15.0 40.0 20.0 0.0 
2. Standards alignment 5.0 20.0 55.0 20.0 0.0 
3. Fidelity to LDC instruction 5.0 20.0 50.0 25.0 0.0 
4. Quality instructional strategies 5.0 15.0 40.0 35.0 5.0 
5. Coherence/clarity of module 10.0 20.0 35.0 30.0 5.0 
6. Overall impression 0.0 10.0 75.0 15.0 0.0 
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Table F9a 


Distribution (Percentage) of Ratings for the Secondary Modules by Cohort 


Dimension 1 2 3 4 5 

COHORT 1, 2016-2017 (n = 14) 

1. Effective writing task 0.0 21.4 35.7 35.7 0.0 
2. Standards alignment 0.0 21.4 35.7 35.7 7.1 
3. Fidelity to LDC instruction 14.3 14.3 35.7 28.6 7.1 
4. Quality instructional strategies 0.0 7.1 21.4 42.9 21.4 
5. Coherence/clarity of module 7.1 7.1 35.7 28.6 21.4 
6. Overall impression 0.0 7.1 50.0 35.7 7.1 
COHORT 1, 2017-2018 (n = 11) 

1. Effective writing task 9.1 9.1 36.4 18.2 9.1 
2. Standards alignment 9.1 18.2 63.6 9.1 0.0 
3. Fidelity to LDC instruction 0.0 27.3 36.4 36.4 0.0 
4. Quality instructional strategies 0.0 18.2 36.4 36.4 9.1 
5. Coherence/clarity of module 9.1 27.3 45.5 9.1 9.1 
6. Overall impression 0.0 18.2 63.6 18.2 0.0 
COHORT 1, 2018-2019 (n = 9) 

1. Effective writing task 0.0 11.1 44.4 22.2 11.1 
2. Standards alignment 0.0 22.2 22.2 44.4 11.1 
3. Fidelity to LDC instruction 11.1 11.1 44.4 22.2 11.1 
4. Quality instructional strategies 11.1 11.1 11.1 55.6 11.1 
5. Coherence/clarity of module 11.1 11.1 44.4 33.3 0.0 
6. Overall impression 0.0 22.2 44.4 33.3 0.0 
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Table F9b 


Distribution (Percentage) of Ratings for the Secondary Modules by Cohort 


Dimension 1 2 3 4 5 

COHORT 2, 2017-2018 (n = 17) 

1. Effective writing task 0.0 11.8 29.4 41.2 17.6 
2. Standards alignment 5.9 11.8 29.4 41.2 11.8 
3. Fidelity to LDC instruction 0.0 5.9 29.4 41.2 11.8 
4. Quality instructional strategies 5.9 5.9 5.9 70.6 5.9 
5. Coherence/clarity of module 0.0 11.8 41.2 35.3 11.8 
6. Overall impression 0.0 11.8 23.5 58.8 5.9 
COHORT 2, 2018-2019 (n = 10) 

1. Effective writing task 10.0 20.0 40.0 20.0 10.0 
2. Standards alignment 10.0 0.0 60.0 20.0 10.0 
3. Fidelity to LDC instruction 20.0 30.0 30.0 20.0 0.0 
4. Quality instructional strategies 0.0 30.0 40.0 20.0 10.0 
5. Coherence/clarity of module 10.0 20.0 30.0 40.0 0.0 
6. Overall impression 0.0 20.0 60.0 20.0 0.0 
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Table F10 


Means and Standard Deviations for the Exploratory Analysis of Elementary Modules by Cohort 


Dimension 2017-2018 2018-2019 Change 

COHORT 1 (n= 4) 

1. Effective writing task -- -- -- 

2. Standards alignment -- -- -- 

3. Fidelity to LDC instruction -- -- -- 

4. Quality instructional strategies -- -- -- 

5. Coherence/clarity of module -- -- -- 

6. Overall impression -- -- -- 
Average (Dimensions 1 to 5) -- -- -- 
COHORT 2 (n = 9) 

1. Effective writing task 4.00 (1.00) 3.44 (0.88) -0.56 (0.73) 
2. Standards alignment 3.78 (1.09) 2.89 (0.93) -0.89 (0.93) 
3. Fidelity to LDC instruction 3.00 (1.41) 3.78 (1.30) 0.78 (1.92) 
4. Quality instructional strategies 3.33 (0.87) 3.44 (1.24) 0.11 (1.36) 
5. Coherence/clarity of module 3.44 (0.88) 3.56 (1.33) 0.11 (1.27) 
6. Overall impression 3.67 (0.71) 3.22 (0.97) -0.44 (1.01) 
Average (Dimensions 1 to 5) 3.51 (0.63) 3.42 (1.01) -0.09 (0.87) 


Note. Means not presented for samples of less than five. 
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Table F11 


Means and Standard Deviations for the Exploratory Analysis of Secondary Modules by Cohort 


Dimension 2017-2018 2018-2019 Change 

COHORT 1 (n = 20) 

1. Effective writing task 3.25 (1.29) 2.75 (1.41) -0.50 (1.54) 
2. Standards alignment 3.00 (1.21) 2.90 (0.91) -0.10 (1.65) 
3. Fidelity to LDC instruction 3.40 (1.27) 3.10 (1.07) -0.30 (1.26) 
4. Quality instructional strategies 4.05 (0.76) 3.65 (1.18) -0.40 (1.35) 
5. Coherence/clarity of module 3.60 (0.82) 3.25 (1.21) -0.35 (1.35) 
6. Overall impression 3.60 (0.75) 3.30 (0.73) -0.30 (1.17) 
Average (Dimensions 1 to 5) 3.46 (0.83) 3.13 (0.83) -0.33 (1.17) 
COHORT 2 (n = 10) 

1. Effective writing task 3.30 (1.16) 3.90 (0.57) 0.60 (1.35) 
2. Standards alignment 3.20 (0.79) 3.50 (0.85) 0.30 (1.06) 
3. Fidelity to LDC instruction 3.70 (1.25) 3.30 (1.42) -0.40 (1.35) 
4. Quality instructional strategies 3.00 (1.15) 3.50 (1.27) 0.50 (1.96) 
5. Coherence/clarity of module 3.30 (0.82) 3.60 (0.70) 0.30 (0.95) 
6. Overall impression 3.00 (0.94) 3.70 (0.67) 0.70 (1.25) 
Average (Dimensions 1 to 5) 3.30 (0.61) 3.56 (0.59) 0.26 (1.00) 
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Table F12 


Distribution (Percentage) of Ratings for the Exploratory Analysis of Elementary Modules by Cohort 


Negative No Change Positive 

Dimension # % # % # % 
COHORT 1 (n = 4) 
1. Effective writing task NA NA NA NA NA NA 
2. Standards alignment NA NA NA NA NA NA 
3. Fidelity to LDC instruction NA NA NA NA NA NA 
4. Quality instructional strategies NA NA NA NA NA NA 
5. Coherence/clarity of module NA NA NA NA NA NA 
6. Overall impression NA NA NA NA NA NA 
Average (Dimensions 1 to 5) NA NA NA NA NA NA 
COHORT 2 (n = 9) 
1. Effective writing task 4 44.4 5 55.6 0 0.0 
2. Standards alignment 7 77.8 1 11.1 1 11.1 
3. Fidelity to LDC instruction 3 33.3 1 11.1 5 55.6 
4. Quality instructional strategies 2 22.2 3 33.3 1 11.1 
5. Coherence/clarity of module 3 33.3 2 22.2 4 44.4 
6. Overall impression 5 55.6 2 22.2 2 22.2 
Average (Dimensions 1 to 5) 4 44.4 1 11.1 4 44.4 
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Table F13 


Distribution (Percentage) of Ratings for the Exploratory Analysis of Secondary Modules by Cohort 


Negative No Change Positive 
Dimension # % # % # % 

COHORT 1 (n = 20) 

1. Effective writing task 10 50.0 4 20.0 6 30.0 
2. Standards alignment 8 40.0 5 25.0 7 35.0 
3. Fidelity to LDC instruction 9 45.0 8 40.0 3 15.0 
4. Quality instructional strategies 7 35.0 8 40.0 5 25.0 
5. Coherence/clarity of module 10 50.0 4 20.0 6 30.0 
6. Overall impression 7 35.0 9 45.0 4 20.0 
Average (Dimensions 1 to 5) 13 65.0 1 5.0 6 30.0 
COHORT 2 (n = 10) 

1. Effective writing task 3 30.0 1 10.0 6 60.0 
2. Standards alignment 2 20.0 5 50.0 3 30.0 
3. Fidelity to LDC instruction 5 50.0 3 30.0 2 20.0 
4. Quality instructional strategies 1 10.0 4 40.0 5 50.0 
5. Coherence/clarity of module 2 20.0 4 40.0 4 40.0 
6. Overall impression 2 20.0 2 20.0 6 60.0 
Average (Dimensions 1 to 5) 2 20.0 2 20.0 6 60.0 
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Appendix G: 
Fidelity Matrix 
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Appendix H: 
Outcome Analysis Methodology 


Analysis Model Specification 


The specification outlined below is for middle school analyses. The elementary model is 
identical with two exceptions. A variable representing the average number of weeks in which a 
student was exposed to core class content was included in the middle school models to capture 
variation in that variable due to students taking extra core science content. Additionally, the 
middle school models include two dummy variables (one for grade 7 and one for grade 8), while 
the elementary school model only required one dummy variable (grade 5). 


For our outcome analyses, we used a threshold of p < .05 to determine whether there was 
a Statistically significant impact of LDC on ELA achievement. In addition to the LDC treatment 
indicator, a teacher effect for years of experience was included, as well as an aggregate 
indicator measuring the mean prior performance of each student’s classroom peers. The fixed 
effects also included student characteristics to identify the matched comparison sample of 
students, such as baseline achievement, socioeconomic status, demographics, language 
proficiency, grade level, and participation in special education. 


The three-level MMMC model was used to estimate the impacts of the LDC intervention 
on student learning. This same analytic model will be used to estimate impacts in future years. 
The general specification for the middle school level MMMC model is shown in the following 
equation using similar notation proposed by Browne et al. (2001, equation 6) and applied in 
Tranmer, Steel, and Browne (2014, equation 3). 


at al (3) (2) 
Vi = XB + Uschooi) y Wily +e 


j€Teacher(i) 


i = 1,...,n Teacher(i) C (1,...,J) 


(3) (2) 
Uschool(i) ~ N(O, Oru(2))s Uy ~ N(0, 0x2), ~ N(0,0¢/) 


In this model y;is the student achievement score response, X;is a vector of the fixed 
covariates and P is the vector of the corresponding fixed effects. School(i) is the school which 
student i attends, thus the term ates represents the random effects for that level of 
classification. Within the term > jereacner(i) wij” ; u” is the set of j random effects for the 
teachers included in the selected dataset, and wj ; is the weight which sums to 1 for each 
student applied in proportion to the instruction time assigned with each teacher. The following 


presents an example of the full model middle school specification. 
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achievement; = By + LDC; * 6, + priorELA; * Bz + priorMath, * Bb, + Female; * 6, + LEP; 
* Bs +SES; * B, + Black; * 6; + Hispanic; * Bg + Asian; * By + OtherEth; 
* By, + SPED; * Biz + Grade7; * B,3 + Grade8; * By4 
+priorClassELA, * Bs + TeachExp; * Bi6 


+ ContentExpWeeks, * Byz + ae ae = wi guy” +e; 


j€Teacher(i) 


(3) (2) 
Uschool(i) ~ N(O, Ou(3))s Uy a N(0, 0x2), i ~ N(0,0¢/) 


In this model, achievement; is the standardized ELA outcome score for student i; In the 
dosage-dependent model LDC; is the proportion of core class instruction time taken with an 
LDC teacher (ranges 0-1); In the dosage-independent model LDC; is coded as zero for 
comparison students and as 1 for students receiving any level of LDC teacher exposure. 


Female;, LEP;, Black;, Hispanic, , Asian;, OtherEth; SPED;,, SES;, Grade7;, Grade8; ar 
e student demographic indicators coded 1 if the status is present and 0 if absent; white 
students serve as the reference group for the race/ethnicity variables, and grade level 6 is the 
reference group for grade level; 


priorELA; , and priorMath, are standardized student achievement scores from the prior 
year; 


priorClassELA; is the aggregated mean of the prior ELA scores for all the core class 
peers of student i; 


TeachExp; is the aggregated percentage of the years of teaching experience less than 3 
for those teachers to which student i was exposed to in her core classes; 


ContentExpWeeks, is the average number of weeks in which student i was exposed to in 
her core classes; 


B, is the impact of LDC, the treatment; 

Biz is the average difference between Cohort 1 and Cohort 2; 
Bz & B3 are the effects of the prior score covariates; 

By ... By4 are the effects of the demographic covariates; 


Bi5& B16 are the effects of the aggregated class level covariates. 


@) (2) e; are the error components at the school, teacher, and student level 


Uschool(i)’ Uj 


respectively assumed to all have mean 0 and variance, O02) O03) og respectively. 
Student/Teacher Course Exposure Weighting 


Tables H1 and H2 demonstrate how the process of calculating general MMMC teacher 
weights and LDC treatment weights (for the dosage modeling approach) was conducted 
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respectively for elementary and middle school. In elementary school, in the event that a 
student was exposed to more than one teacher, each content area was given equal weight in 
distributing teacher/student exposure. For example, if a student was enrolled for both ELA and 
social studies/history under one teacher, then that teacher was coded as .67 for having 
contributed to two thirds of the students’ core curriculum exposure. If the same student 
enrolled in science with a different teacher than the one who was linked to their course marks 
in ELA and social studies/history, then that science teacher would have been coded as .33 and 
all other teachers in the sample would have been coded as zero. This would then result in the 
student’s exposure adding to a unity (1). 


Table H1 
Example of Elementary School Student/Teacher Weighting Based on Course Links 


Example student/teacher weighting for use in Example LDC 


Core content MMMC treatment 
area (Weight=Subject Days/Total Days) dosage weight 

ELA Student enrolled Weight=.333 
with an intervention Teacher: 
Weight=0.333 

Social studies Student enrolled Weight=.333 
with an intervention Teacher: 
Weight=0.333 

Science Student enrolled Weight=.0 
with a non-intervention teacher: 
Weight=0.333 

Total Unity: for every student the student/teacher Treatment 
weights sum to 1 Weight=.667 
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Table H2 
Example of Middle School Student/Teacher Weighting Based on Course Mark Links 


Example student/teacher weighting for use in Example LDC 


Core content MMMC treatment 
area Course name (Weight=Subject Days/Total Days) dosage weight 
ELA English 7A & 7B Student enrolled two hundred days of core ELA Weight=.286 


(grade 7) with an intervention teacher: 
Weight=200/700=0.286 


Social studies Social Studies WHG: Student enrolled two hundred days of core Weight=.0 
ANCCIV &B social studies (grade 7) with a non-intervention 
teacher: 


Weight=200/700=0.286 
Science Science 7 Student enrolled three hundred days of core Weight=.0 


science (grade 7) with a non-intervention 
teacher: 


Weight=300/700=0.428 


Total Unity: for every student the student/teacher Treatment 
weights sum to 1 Weight=.286 


Note. “Days” refers to core content enrolled days preceding the NYS assessment date. 


In middle school, students’ exposure to teachers at the course level in the three core 
content areas was coded in the same manner as in the elementary grades based on enrolled 
time preceding the assessment period. A difference in our middle school coding process was 
that we did not force each core content area into equal weighting. Instead each core content 
area exposure contributed to a core content area total sum which formed the basis from which 
the weights were proportioned. 


Most commonly a student had equivalent days of core instruction exposure in each of the 
three content areas (often 215 days in each content area). In that scenario, if a student had 
exposure to three different teachers, then each teacher would contribute one third (.33) of the 
overall core curriculum exposure and all other teachers in the sample would be coded as zero. 
However, in addition to the typical core science course, extra core science courses were also 
included in the LDC analysis (for example a Grade 8 student taking biology), which made it 
possible then for a student to accumulate more units in science than in the other two content 
areas. 


The weighting in middle school was always distributed as a proportion of the total 
exposure days in the three content areas. Therefore, if a student accumulated 300 science days 
(across two courses), 200 social studies days, and 200 ELA days, the base number of instruction 
days would be 700 days. If, using that same scenario, the same teacher taught both the typical 
core and biology courses then that teacher would contribute three-sevenths (.43) of the overall 
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core curriculum exposure with the social studies and science teachers contributing two- 
sevenths (.285) each, again resulting in the student’s exposure adding to a unity (1). 


Calculation of Effect Size 


We calculated student-level effect sizes according to the WWC 3.0 criteria. Specifically, for 
the impact analysis with treatment status as a dichotomous variable, we calculated Hedges’ g, 
the difference in adjusted mean outcomes for the groups divided by the unadjusted pooled 
within-group standard deviation of the outcome measure in the sample, for all outcomes. The 
difference in adjusted mean outcomes is estimated by £, in the models defined above, as 
outlined in the WWC standards handbook for computing effect sizes in multi-level frameworks. 
Specifically, 


wy 

SOS 

(nj - 1)s? + (n, - 1)? 
ni tn, — 2 


where y is f, which is the coefficient from the MMMC for the intervention effect. 


Note that in our analyses the outcome measure is standardized within the analytical 
sample (mean=0, standard deviation=1). As a result, we expected that g would likely be quite 
similar to the 6, coefficient from the MMMC Model in the large samples we plan to collect later 
in the study. 


Though it is not standard to use g with a continuous treatment effect, as in the case of 
our dosage-dependent treatment measure, we have defined and matched populations n; and 
n, where treated students (n;) could have any positive treatment value<=1, and control 
students (n,) had a treatment value of zero. We could therefore calculate g in the case of our 
dosage-dependent treatment measure, and again expect that it would not differ substantially 
from the f, coefficient. It is crucial, however, to note that g and f, in the dosage-dependent 
models reflect the effect size projected for a student who would receive exposure to 
intervention teachers in all of their core classes. Along with this effect we report average 
dosage received by treated students so that the average treatment effect on treated students 
could be calculated. 
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Appendix I: 
Supplemental Outcome Analysis Tables 


In this appendix, we provide tables displaying the sample sizes and matching results for all 
the supplementary analyses. We also provide full model results for the supplementary analyses 
which divide the primary treatment sample into groups based on prior student exposure to 
LDC, and the amount of exposure in the outcome year. Here is the full list of supplementary 
analyses contained in this appendix: 


1.1 2016-2017 Cohort 1 Elementary School Sample (1 year) 
1.2 2016-2017 Cohort 1 Middle School Sample (1 year) 


1.3 2017-2018 Cohort 1 Middle School Sample (Participated in 2016-2017 and 2017— 
2018) 


1.4 2017-2018 Cohort 1 Middle School Sample (Newly Joined LDC in 2017-2018) 


1.5 2018-2019 Cohort 1 Middle School Sample (Began Participating in 2017-2018 and 
Continued Through 2018-2019) 


1.6 2017-2018 Cohort 2 Elementary School Sample (1 year) 

1.7 2018-2019 Cohort 2 Elementary School Sample (2 years) 

1.8 2017-2018 Cohort 2 Middle School Sample (1 year) 

1.9 2018-2019 Cohort 2 Middle School Sample (2 years) 

1.10 2018-2019 Cohort 2 Middle School Sample (Newly Joined LDC in 2018-2019) 


1.11 Student Exposure Analysis 


1.1 2016-2017 Cohort 1 Elementary School Sample (1 year) 


The 2016-2017 elementary sample prior to the CEM process included 5 schools, the 15 
elementary teachers who consented to participate in the evaluation study, and their 273 
students. After the student-level matching, our final elementary LDC sample included 234 
elementary students and retained nearly the same number of teachers and schools (see Table 
11). Prior to matching, the potential elementary comparison sample consisted of 781 schools, 
10,106 teachers, and 123,307 students. This comparison sample was substantially reduced 
during the first stage of matching, which identified the five schools that most closely resembled 
each of the 5 LDC elementary schools. After student-level CEM, a workable analytic comparison 
sample consisted of 19 schools, 100 teachers and 234 students. 
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Table |1 
Before and After Matching Sample Sizes: 2016—2017 Cohort 1 Elementary School Analysis 


LDC sample Comparison sample 
School Teacher Student School Teacher Student 
Stage 1 
Before matching 5 15 273 781 10,106 123,307 
After matching 5 15 273 25 474 4,979 
Stage 2 
After matching 5 14 234 19 100 234 


Table I2 presents the student characteristics for the LDC student and comparison 
students, based on the final analytical samples for the elementary analyses. Treatment and 
comparison samples matched very closely on all variables. Exact matching was achieved on 
some variables, and all demographic variables were within 4 percentage points. The final LDC 
elementary student sample after matching was comprised largely of Black, Asian, and Hispanic 
students, with more than one third of being of Black ethnicity, almost one third Asian, about 
slightly over one quarter Hispanic. The sample was comprised of fourth and fifth grade students 
and mostly of students who were economically disadvantaged (88.5%). English language 
learners and special education students each represented between 10 and 15 percent of the 
sample. Mean performance on the prior year academic assessment was very close to the 
district-wide performance levels in mathematics and ELA. 
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Table 12 


2015-2016 Baseline Characteristics of Treated Students Taught by Cohort 1 Elementary School Teachers 
in 2016-2017 and Comparison Students After Matching 


Student characteristic LDC sample: Comparison sample: 


(n = 234) (n = 234) 


Race/Ethnicity 


Hispanic (%) 25.6 27.8 
Black (%) 38.0 34.6 
Asian (%) 30.8 30.8 
White (%) 3.0 4.3 
Other (%) 2.6 2.5 
Female (%) 47.9 48.7 


Special programs status 


Poverty (%) 88.5 90.4 
English Language Learner (%) 12.4 8.5 
Special education (%) 14.1 12.8 


Student baseline achievement 
Mean mathematics Z score -0.01 -0.03 
Mean ELA Z score 0.01 0.02 


Class & teacher characteristics 


Mean ELA Z score of current peers 0.02 -0.02 

Teacher years of experience 11.5 11.9 
Grade level 

Grade 3 at baseline (%) 62.4 62.4 

Grade 4 at baseline (%) 37.6 37.6 


Table I3 reports the model results for the effect of LDC teachers in 2016-2017 on student 
outcomes. The outcome variable for both models was students’ 2017 New York State 
assessment scores in ELA. For technical reasons of evaluating effect sizes, New York State 
assessment scale scores were standardized to the study sample. Most treated students were 
only exposed to one teacher (the LDC intervention teacher); thus, the average treated student 
in this sample received a 0.72 treatment dosage. As reported, the dosage dependent effect is in 
the positive direction and the dosage independent effect is in the negative direction, but 
neither effect was statistically significant. In other words, neither model provided sufficient 
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evidence to conclude that students taught by LDC teachers performed at levels different than 
their matched peers in the comparison group. 


The effects of the covariates on student performance were similar under both models and 
were in the expected directions (although most were not statistically significant). Prior ELA 
performance was the strongest predictor and prior mathematics performance also helped 
explain the outcome. In addition to prior achievement, one demographic variable (gender) 
helped predict performance. Females performed at significantly higher levels than males. 


Table 13 


Effect Estimates of Cohort 1 Elementary School Teachers Participating in LDC for One Year on 2016-2017 
New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables Dosage Dependent Dosage independent 
model coefficient (SD) model coefficient (SD) 


Level 1 - LDC teacher treatment 
Level 2 - Student characteristics 

Hispanic 

Black 

Asian 

Other 

Poverty 

Female 

ELL 

Special Education 

Grade 4 at baseline 

Teacher experience 


Baseline peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


0.084 (0.147) 


-0.110(0.163) 
-0.195 (0.152) 
-0.143 (0.148) 
-0.306 (0.208) 
-0.058 (0.065) 
0.171 (0.053)* 
-0.034 (0.107) 
-0.169 (0.089) 
-0.121 (0.105) 
0.001 (0.010) 
0.152 (0.092) 
0.166 (0.044)* 
0.528 (0.051)* 


-0.059 (0.118) 


-0.113(0.163) 
-0.191 (0.152) 
-0.144 (0.148) 
-0.302 (0.208) 
-0.055 (0.065) 
0.169 (0.053)* 
-0.030 (0.107) 
-0.168 (0.089) 
-0.126 (0.104) 
0.001 (0.010) 
0.144 (0.092) 
0.167 (0.044)* 
0.528 (0.051)* 


Note. Since the average treatment student received a 0.724 treatment dosage we could estimate an 
average treatment effect on the treated (ATET) at 0.724 * 0.084 = 0.061. This effect was not statistically 
significant. Inclusion rules led to removal of ELL students and students with disabilities variables from 
the model. 

*Cl two tailed probability > .95 
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1.2 2016-2017 Cohort 1 Middle School Sample (1 year) 


The 2016-2017 LDC middle school sample included twenty schools, 104 teachers, and 
3,562 students prior to the CEM process. After the CEM student-level matching, our final 
secondary LDC sample was reduced to 3,214 students, with all schools and teachers being 
retained (see Table 14). Prior to matching, the potential comparison sample consisted of 519 
schools, 10,131 teachers and 161,047 students. This comparison sample was substantially 
reduced during the first stage of matching, which identified the five schools that most closely 
resembled each of the 20 LDC schools. After student-level CEM, an analytic comparison sample 
of 85 schools, 1,423 teachers, and 3,214 students was constructed. 


Table 14 
Before and After Matching Sample Sizes: 2016-2017 Cohort 1 Middle School Analysis 


LDC sample Comparison sample 
Schools Teachers Students Schools Teachers Students 
Stage 1 
Before matching 20 104 3,562 519 10,131 161,047 
After matching 20 104 3,562 100 2,304 29,375 
Stage 2 
After matching 20 104 3,214 85 1,423 3,214 


As shown in Table 15, the final LDC middle school student sample included a large 
proportion of Black and Hispanic students (87.2%), and a similarly large proportion of students 
who were economically disadvantaged (86.1%). The sample was comprised of sixth (16.1%), 
seventh (41.1%), and eighth grade students (42.8%). Special education students represented 
slightly less than one-quarter of this sample, while there was about one half that many English 
language learners (12.8%). In addition, mean performance on the prior year academic 
assessment was somewhat lower for LDC students as compared to district-wide performance 
levels in mathematics and ELA. Treatment and comparison samples matched very closely on all 
variables. Exact matching was achieved on some variables, and all demographic variables were 
within 3 percentage points. 


225 


Table I5 


2015-2016 Baseline Characteristics of Treated Students Taught by Cohort 1 Middle School Teachers in 
2016-2017 and Comparison Students After Matching 


Student characteristic LDC sample: Comparison sample: 


(n = 3,214) (n = 3,214) 


Race/Ethnicity 


Hispanic (%) 56.1 57.1 
Black (%) 31.1 29.8 
Asian (%) 5.2 6.4 
White (%) 6.7 5.7 
Other (%) 0.9 0.9 
Female (%) 51.8 48.9 


Special programs status 


Poverty (%) 86.1 86.4 
English Language Learner (%) 12.8 13.5 
Special education (%) 23.6 25.8 


Student baseline achievement 
Mean math Z score -0.40 -0.44 
Mean ELA Z score -0.38 -0.37 


Class & teacher characteristics 


Mean ELA Z score of current peers -0.41 -0.37 

Teacher years of experience 8.7 8.4 
Grade level 

Grade 5 at baseline (%) 16.1 16.1 

Grade 6 at baseline (%) 41.1 41.1 

Grade 7 at baseline (%) 42.8 42.8 


In Table I6 we present results of both the dosage dependent and dosage independent 
models on middle school students’ ELA performance in 2016-2017 after being taught by 
teachers in their first year of LDC implementation. Model results show no statistically 
discernible LDC effect on student outcomes for either model. In other words, both analyses 
showed that students taught by LDC teachers scored similarly on the ELA test to their matched 
peers in the comparison group. The effects of the covariates on student performance were 
similar in direction to those in the elementary school models for the same cohort and outcome 
year, although due to the larger sample size more demographic variables were significant in the 


226 


middle school analyses. Prior ELA and mathematics performance, as well as ELL status, special 
education status, Black ethnicity, Other Ethnicity, and gender were significant predictors of ELA 


performance. In secondary schools, the average prior achievement of a student’s peers was 
also a significant predictor of ELA performance in the expected direction. 


Table |6 


Effect Estimates of Cohort 1 LDC Middle School Teachers Participating in LDC for One Year on 2016-2017 
New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables 


Dosage-dependent model 


coefficient (SD) 


coefficient (SD) 


Dosage-independent model 


Level 2 - LDC teacher treatment 
Level 1 - student characteristics 
Hispanic 
Black 
Asian 
Other 
Poverty 
Female 
English language learner 
Special education 
Grade 6 at baseline 
Grade 7 at baseline 
Content Exposure Weeks 
Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


-0.016 (0.066) 


-0.069 (0.036) 
-0.126 (0.038)* 
0.047 (0.046) 
-0.165 (0.083)* 
-0.039 (0.022) 
0.094 (0.015)* 
-0.101 (0.027)* 
-0.147 (0.020)* 
0.054 (0.032) 
0.108 (0.035)* 
-0.003 (0.002) 
-0.005 (0.003) 
0.155 (0.030)* 
0.179 (0.012)* 
0.658 (0.013)* 


-0.019 (0.042) 


-0.069 (0.036) 
-0.126 (0.038)* 
0.047 (0.046) 
-0.165 (0.083)* 
-0.039 (0.022) 
0.094 (0.015)* 
-0.101 (0.027)* 
-0.147 (0.020)* 
0.054 (0.032) 
0.108 (0.035)* 
-0.003 (0.002) 
-0.005 (0.003) 
0.155 (0.030)* 
0.179 (0.012)* 
0.658 (0.013)* 


Note. Since the average treatment student received a 0.506 treatment dosage we could estimate an 


average treatment effect on the treated (ATET) at 0.506 * -0.016 = -0.008. This effect was not 


statistically significant. 
*Cl two tailed probability > .95 
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1.3 2017-2018 Cohort 1 Middle School Sample (Participated in 2016-2017 and 
2017-2018) 


The 2017-2018 Cohort 1 middle school sample (teachers who participated in LDC in 
2016-2017 and 2017-2018) prior to the CEM process included ten schools, 26 Cohort 1 
returning middle school teachers who consented to participate in the evaluation study, and 
their 1,626 students. After the student-level matching, the final LDC sample included 1,511 
students and the same number of teachers and schools prior to matching (see Table I7). Prior to 
matching, the potential middle school comparison sample consisted of 514 schools, 9,871 
teachers, and 151,030 students. This comparison sample was substantially reduced during the 
first stage of matching, which identified up to five schools that closely resembled each of the 
ten LDC middle schools. After student-level CEM, a workable analytic comparison sample 
consisted of 43 schools, 753 teachers and 1,511 students. 


Table 17 


Before and After Matching Sample Sizes: Cohort 1 Middle School Teachers Participating in LDC in 2016- 
2017 and 2017-2018 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before matching 10 26 1,475 514 9,871 151,030 
After matching 10 26 1,475 45 896 12,177 
Stage 2 
After matching 10 26 1,408 44 702 1,408 


The final LDC Cohort 1 middle school student sample after matching was comprised 
largely of students who were economically disadvantaged (see Table 18). Slightly more than half 
of students were Hispanic and about a third were black, and the sample was well distributed 
across the three grade levels in the baseline year. Special education students represented 
about one-quarter of this sample, while English language learners represented slightly more 
than 10%. Mean performance on the prior year assessments was more than one-third of a 
standard deviation lower than the district-wide performance levels in both mathematics and 
ELA. 
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Table 18 


2015-2016 Baseline Characteristics of Treated Students Taught by Cohort 1 Middle School Teachers 
Participating in LDC in 2016-2017 and 2017-2018 and Comparison Students After Matching 


Treatment group Comparison group 
Student characteristic (n = 1,408) (n = 1,408) 

Race/ethnicity 

Hispanic (%) 52.4 52.4 

Black (%) 31.9 31.9 

Asian (%) 3.9 3.9 

White (%) 10.9 10.6 

Other (%) 0.9 1.2 
Female (%) 54.3 54.3 
Special programs status 

Poverty (%) 85.6 84.9 

English language learner (%) 12.6 12.9 

Special education (%) 26.0 23.7 
Student prior achievement 

Mean baseline year mathematics Z score -0.412 -0.391 

Mean baseline year ELA Z score -0.341 -0.345 
Class & teacher characteristics 

Mean baseline ELA Z score of current peers -0.373 -0.329 

Mean Teacher years of experience 7.5 7.1 
Grade level at baseline year 

Grade 4 at baseline (%) 27.7 27.7 

Grade 5 at baseline (%) 39.7 39.7 

Grade 6 at baseline (%) 32.6 32.6 


In Table 19, we present results model results for the effect of LDC as practiced by Cohort 1 
middle school teachers for two consecutive years on 2017-2018 ELA Assessment scores. 
Student outcomes are in the positive direction, but are not statistically significant for either 
model. In other words, neither analysis provided sufficient evidence to conclude that students 
taught by LDC teachers performed better on the ELA test than did their matched peers in the 
comparison group. 
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Table 19 


Effect Estimates of Cohort 1 Middle School Teachers Participating in LDC in Two Consecutive Years on 
2017-2018 New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables Dosage-dependent model Dosage-independent model 
coefficient (SD) coefficient (SD) 


Level 2 - LDC teacher treatment 
Level 1 - student characteristics 

Hispanic 

Black 

Asian 

Other 

Poverty 

Female 

English language learner 

Special education 

Grade 5 at baseline 

Grade 6 at baseline 

Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


0.001 (0.133) 


-0.084 (0.049) 
-0.180 (0.053)* 
0.110 (0.073) 
0.147 (0.123) 
-0.114 (0.035)* 
0.195 (0.025)* 

-0.008 (0.040) 

-0.104 (0.032)* 
0.107 (0.042)* 
0.112 (0.045)* 
0.002 (0.004) 

0.284 (0.035)* 
0.179 (0.019)* 
0.579 (0.021)* 


-0.009 (0.056) 


-0.084 (0.049) 
-0.180 (0.054)* 
0.109 (0.073) 
0.147 (0.123) 
-0.114 (0.035)* 
0.195 (0.025)* 

-0.008 (0.040) 

-0.104 (0.032)* 
0.108 (0.042)* 
0.112 (0.045)* 
0.002 (0.004) 

0.283 (0.035)* 
0.179 (0.019)* 
0.579 (0.021)* 


Note. Based on the dosage-dependent model, the average treated student received a 0.344 treatment 
dosage. Because of this, using the dosage dependent model we estimate average treatment effect on 
the treated (ATET) at (0.345 * 0.001) = 0.0003. 

*Cl two tailed probability > .95 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Prior ELA performance was the strongest 
predictor and prior mathematics performance also helped explain the outcome. In addition to 
prior achievement, a number of demographic variables helped predict performance: Black 
students performed at lower levels than White students, students in special education 
performed at lower levels than students not in special education, females performed at 
significantly higher levels than males, and students with low socio-economic status performed 
at lower levels than did their peers with higher socio-economic status. 
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1.4 2017-2018 Cohort 1 Middle School Sample (Newly Joined LDC in 2017-2018) 


The 2017-2018 Cohort 1 middle school sample (newly joined teachers with 1 year of LDC 
participation) prior to the CEM process included seven schools, 17 Cohort 1 middle school 
teachers who consented to participate in the evaluation study, and their 1,023 students. After 
the student-level matching, our LDC sample included 941 students and the same number of 
teachers and schools prior to matching (see Table 110). Prior to matching, the potential middle 
school comparison sample consisted of 514 schools, 9,871 teachers, and 151,030 students. This 
comparison sample was substantially reduced during the first stage of matching, which 
identified up to five schools that closely resembled each of the seven LDC middle schools. After 
student-level CEM, a workable analytic comparison sample consisted of 29 schools, 487 
teachers and 941 students. 


Table 110 
Before and After Matching Sample Sizes: 2017-2018 Cohort 1 Middle School Analysis (Newly Joined LDC 
in 2017-2018) 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before matching 7 17 1,023 514 9,871 151,030 
After matching 7 17 1,023 31 722 7,931 
Stage 2 
After matching 7 17 941 29 487 941 


The final LDC sample after matching was comprised largely of students who were 
economically disadvantaged (see Table 111). Over two thirds of students were Hispanic and a 
little over a quarter were black, and the sample was well distributed across the three grade 
levels in the baseline year. Special education students represented about one-quarter of this 
sample, while English language learners represented about one eighth. Mean performance on 
the prior year assessments was about one-half of a standard deviation lower than the district- 
wide performance levels in both mathematics and ELA. 
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Table 111 


2015-2016 Baseline Characteristics of Students Taught by Cohort 1 Middle School Teachers Who Newly 
Joined LDC in 2017-2018 and Comparison Students After Matching 


Treatment group 


Comparison group 


Student characteristic (n = 941) (n = 941) 

Race/ethnicity 

Hispanic (%) 70.5 70.5 

Black (%) 27.1 27.1 

Asian (%) 1.0 1.0 

White (%) 1.2 1.3 

Other (%) 0.3 0.2 
Female (%) 49.3 49.3 
Special programs status 

Poverty (%) 91.8 91.8 

English language learner (%) 12.8 12.8 

Special education (%) 26.7 26.2 
Student baseline achievement 

Mean baseline year mathematics Z score -0.527 -0.499 

Mean baseline year ELA Z score -0.419 -0.409 
Class & teacher characteristics 

Mean baseline ELA Z score of current peers -0.446 -0.442 

Teacher years of experience 8.6 8.3 
Grade level at baseline year 

Grade 4 at baseline (%) 39.7 39.7 

Grade 5 at baseline (%) 29.5 29.5 

Grade 6 at baseline (%) 30.7 30.7 


In Table 112 we present the analysis results for students taught by Cohort 1 teachers who 
newly joined existing LDC PLCs in 2017-2018 (these teachers had 1 year of LDC experience but 
their schools were in their second year of LDC implementation). As can be seen, the LDC effect 


on student ELA outcomes are in the negative direction, but are not statistically significant for 


either model. In other words, neither model provided sufficient evidence to conclude that 
students taught by LDC teachers performed differently on the ELA test than did their matched 


peers in the comparison group. 
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Table 112 


Effect Estimates of Cohort 1 Middle School Teachers Joining LDC in 2017-2018 on 2017-2018 New York 
State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables Dosage-dependent model Dosage-independent model 
coefficient (SD) coefficient (SD) 


Level 2 - LDC teacher treatment -0.120 (0.204) -0.061 (0.077) 


Level 1 - student characteristics 


Hispanic 

Black 

Asian 

Other 

Poverty 

Female 

English language learner 
Special education 

Grade 5 at baseline 
Grade 6 at baseline 
Content Exposure Weeks 
Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


-0.060 (0.138) 
-0.167 (0.142) 
0.240 (0.208) 
-0.610 (0.325) 
-0.006 (0.056) 
0.231 (0.031)* 
-0.043 (0.053) 
-0.105 (0.042)* 
0.052 (0.059) 
0.138 (0.062)* 
-0.009 (0.008) 
-0.007 (0.006) 
0.240 (0.046)* 
0.175 (0.026)* 
0.540 (0.027)* 


-0.060 (0.139) 
-0.168 (0.142) 
0.239 (0.207) 
-0.604 (0.323) 
-0.005 (0.056) 
0.231 (0.031)* 
-0.043 (0.053) 
-0.104 (0.041)* 
0.053 (0.059) 
0.142 (0.062)* 
-0.009 (0.008) 
-0.007 (0.006) 
0.240 (0.046)* 
0.174 (0.026)* 
0.540 (0.027)* 


Note. Based on the dosage-dependent model, the average treated student received a 0.281 treatment 
dosage. Because of this, using the dosage dependent model we estimate an average treatment effect on 
the treated (ATET) at (0.281 *- 0.120) = -0.034. 

*Cl two tailed probability > .95 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Prior ELA performance was the strongest 
predictor and prior mathematics performance also helped explain the outcome. In addition to 
prior achievement, gender and special education status also helped predict the outcome. 


233 


1.5 2018-2019 Cohort 1 Middle School Sample (Began Participating in 2017- 
2018 and Continued Through 2018-2019) 


The 2018-2019 Cohort 1 middle school sample (began participating in 2017-2018 and 
continued through 2018-2019) prior to the CEM process included two schools, 8 Cohort 1 
middle school teachers who consented to participate in the evaluation study, and their 552 
students. After the student-level matching, our LDC sample included 500 students and the same 
number of teachers and schools prior to matching (see Table 113). Prior to matching, the 
potential middle school comparison sample consisted of 480 schools, 9,965 teachers, and 
153,981 students. This comparison sample was substantially reduced during the first stage of 
matching, which identified up to five schools that closely resembled each of the two LDC middle 
schools. After student-level CEM, a workable analytic comparison sample consisted of 9 
schools, 155 teachers and 500 students. 


Table 1.13 


Before and After Matching Sample Sizes: 2018-2019 Cohort 1 Middle School Analysis (Joined in 2017- 
2018 and Continued in 2018-2019) 


LDC sample Comparison sample 
Stage School Teacher Student School Teacher Student 
Stage 1 
Before matching 2 8 552 480 9,965 153,981 
After matching 2 8 552 9 170 2,189 
Stage 2 
After matching 2 8 500 9 155 500 


The final LDC sample after matching was comprised largely of students who were 
economically disadvantaged (see Table 114). About two thirds of students were Hispanic and 
about one third were black. Special education students represented about one-quarter of this 
sample, while English language learners represented a little over 10%. Mean performance on 
the prior year assessments was over one-third of a standard deviation lower than the district- 
wide performance levels in both mathematics and ELA. 
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Table 114 


2015-2016 Baseline Characteristics of Students Taught by Cohort 1 Middle School Teachers Who Joined 
LDC in 2017-2018 and Continued Through 2018-2019 and Comparison Students After Matching 


Treatment group Comparison group 
Student characteristic (n = 500) (n = 500) 

Race/ethnicity 

Hispanic (%) 65.8 65.8 

Black (%) 30.6 30.6 

Asian (%) 0.6 2.2 

White (%) 2.8 12 

Other (%) 0.2 0.2 
Female (%) 48.4 44.6 
Special programs status 

Poverty (%) 88.2 85.6 

English language learner (%) 10.2 12.2 

Special education (%) 23.8 23.8 
Student prior achievement 

Mean baseline year mathematics Z score -0.457 -0.475 

Mean baseline year ELA Z score -0.378 -0.381 
Class & teacher characteristics 

Mean baseline ELA Z score of current peers -0.412 -0.397 

Teacher years of experience 11.2 10.1 
Grade level at baseline year 

Grade 3 at baseline (%) 20.6 20.6 

Grade 4 at baseline (%) 35.6 35.6 

Grade 5 at baseline (%) 43.8 43.8 


In Table 115, we present the analysis results for those students taught by Cohort 1 middle 
school teachers who began participating in 2017-2018 and continued through 2018-2019. 
Model results for the LDC effect on student outcomes are in the positive direction, but are not 
Statistically significant for either model. In other words, neither analysis provided sufficient 
evidence to conclude that students taught by LDC teachers performed better on the ELA test 
than did their matched peers in the comparison group. 
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Table 115 


Effect Estimates of Cohort 1 Middle School Teachers Who Began Participating in LDC in 2017-2018 on 
2018-2019 New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables Dosage-dependent model Dosage-independent model 
coefficient (SD) coefficient (SD) 


Level 2 - LDC teacher treatment 
Level 1 - student characteristics 
Hispanic 
Black 
Poverty 
Female 
English language learner 
Special education 
Grade 4 at baseline 
Grade 5 at baseline 
Teacher Experience 
Baseline Peer ELA Z score 
Baseline year mathematics Z score 


Baseline year ELA Z score 


0.421 (0.297) 


-0.019 (0.118) 
-0.092 (0.126) 
0.125 (0.064) 
0.231 (0.044)* 
0.174 (0.076)* 
-0.102 (0.065) 
0.078 (0.107) 
0.069 (0.107) 
-0.002 (0.010) 
0.394 (0.075)* 
0.148 (0.038)* 
0.513 (0.039)* 


0.133 (0.139) 


-0.022 (0.118) 
-0.093 (0.126) 
0.125 (0.064) 
0.232 (0.044)* 
0.170 (0.076)* 
-0.101 (0.065) 
0.090 (0.108) 
0.067 (0.108) 
-0.002 (0.010) 
0.392 (0.075)* 
0.149 (0.038)* 
0.513 (0.039)* 


Note. Based on the dosage-dependent model, the average treated student received a 0.355 treatment 
dosage. Because of this, using the dosage dependent model we estimate an average treatment effect on 
the treated (ATET) at (0.355 * 0.421) = 0.149. 

*Cl two tailed probability > .95 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected directions. Prior ELA performance was the strongest 
predictor and prior mathematics performance also helped explain the outcome. In addition to 
prior achievement, females performed at significantly higher levels than males. One interesting 
result was that English language learners in this analysis were found to have statistically 
significant higher ELA scores than the non-ELL students. 


1.6 2017-2018 Cohort 2 Elementary School Sample (1 year) 


As reported in Table 116, the resulting Cohort 2 elementary LDC one-year sample included 
14 schools, 57 teachers, and 1,351 students prior to the CEM process. After the CEM student- 
level matching, our final Cohort 2 elementary LDC sample was reduced to 54 teachers and 
1,258 students. Prior to matching, the potential comparison sample consisted of 781 schools, 
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10,722 teachers and 120,696 students. This comparison sample was substantially reduced 
during the first stage of matching, which identified up to five schools that most closely 
resembled each of the 14 LDC schools. After student-level CEM, a workable analytic comparison 
sample of 65 schools, 538 teachers, and 1,258 students was constructed. 


Table 116 
Before and After Matching Sample Sizes: 2017-2018 Cohort 2 Elementary School Analysis (1 year) 


LDC sample Comparison sample 
Stage School Teacher Student School Teacher Student 
Stage 1 
Before matching 14 57 1,351 781 10,722 120,696 
After matching 14 57 1,351 70 1,962 9,722 
Stage 2 
After matching 14 54 1,258 65 538 1,258 


The final Cohort 2 LDC elementary student sample (see Table 117) was also more than one 
half Hispanic and about a third black. Most students were identified as economically 
disadvantaged. The sample was quite evenly distributed between students in third (51.8%) and 
fourth grade (48.2%) in the prior year. English language learners represented slightly more than 
15% of this sample, while special education students represented slightly less than one quarter 
of the sample. In addition, mean performance on the prior year ELA assessments was about 
one-quarter of a standard deviation lower than the district-wide performance level, while mean 
performance on the prior year mathematics assessment was about one-third of a standard 
deviation lower than the district-wide performance level. 
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Table 117 


2016-2017 Baseline Characteristics of Treated Students Taught by Cohort 2 Elementary School Teachers 
in 2017-2018 and Comparison Students After Matching 


Treatment group Comparison group 
Student characteristic (n = 1,258) (n = 1,258) 

Race/ethnicity 

Hispanic (%) 56.0 56.0 

Black (%) 35.6 35.6 

Asian (%) 3.0 3.5 

White (%) 3.8 3.7 

Other (%) 1.5 1.2 
Female (%) 50.9 52.6 
Special programs status 

Poverty (%) 85.0 86.3 

English language learner (%) 15.3 12.2 

Special education (%) 22.9 20.4 
Student baseline achievement 

Mean baseline year mathematics Z score -0.320 -0.331 

Mean baseline year ELA Z score -0.247 -0.237 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.244 -0.245 

Teacher years of experience 12.7 12.2 
Grade level at baseline year 

Grade 3 at baseline (%) 51.8 51.8 

Grade 4 at baseline (%) 48.2 48.2 


Table 118 presents results of both the dosage dependent and dosage independent models 
on Cohort 2 elementary school students’ ELA performance in 2017-2018 after being taught by 
teachers with 1 year of LDC experience. The LDC effect on student outcomes are in the negative 
direction, but are not statistically significant for either model. In other words, neither analysis 
provided sufficient evidence to conclude that students taught by LDC teachers performed at 
levels different than their matched peers in the comparison group. The effects of the covariates 
on student performance were in the expected directions. 
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Table 118 


Effect Estimates of Cohort 2 LDC Elementary School Teacher Participating in LDC for One Year on 2017- 
2018 New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables Dosage Dependent 
model coefficient (SD) 


Dosage independent 
model coefficient (SD) 


Level 1 - LDC teacher treatment 
Level 2 - Student characteristics 
Hispanic 
Black 
Asian 
Other 
Poverty 
Female 
English Language Learner 
Special education 
Grade 4 at baseline 
Teacher years of experience 


Baseline peer ELA Z score 


-0.045 (0.067) 


-0.077 (0.073) 
-0.111 (0.074) 
0.014 (0.094) 
-0.030 (0.121) 
-0.057 (0.039) 
0.079 (0.025) * 
-0.160 (0.040) * 
-0.154 (0.035) * 
0.008 (0.040) 
0.001 (0.004) 
0.100 (0.039) * 


-0.003 (0.050) 


-0.077 (0.073) 
-0.111 (0.074) 
0.014 (0.094) 
-0.031 (0.121) 
-0.057 (0.039) 
0.079 (0.025) * 
-0.161 (0.040) * 
-0.154 (0.035) * 
0.012 (0040) 
0.001 (0.004) 
0.100 (0.039) * 


0.234 (0.022) * 0.234 (0.022) * 
0.593 (0.022) * 0.593 (0.022) * 


Baseline year mathematics Z score 


Baseline year ELA Z score 


Note. Based on the dosage-dependent model, the average treated student received a 0.642 treatment 
dosage. Because of this, using the dosage-dependent model we could estimate an average treatment 
effect on the treated (ATET) at (0.642 * -0.045) = -0.029. 

*Cl two tailed probability > .95 


1.7 2018-2019 Cohort 2 Elementary School Sample (2 years) 


Prior to the CEM process, the Cohort 2 elementary two-year LDC sample included 9 
schools, 17 teachers, and 356 students. After the student-level matching, our final pooled 
elementary LDC sample included 341 students and the same number of teachers and schools 
prior to matching (see Table 119). Prior to matching, the potential elementary school 
comparison sample consisted of 776 schools, 6,124 teachers, and 56,083 students. This 
comparison sample was substantially reduced during the first stage of matching, which 
identified up to five schools that closely resembled each of the nine LDC schools. After student- 
level CEM, a workable analytic comparison sample consisted of 40 schools, 221 teachers, and 
341 students. 
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Table 119 


Before and After Matching Sample Sizes: Cohort 2 Elementary School Analysis for Teachers with 2 years 
of LDC Implementation 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 9 17 356 776 6,124 56,083 
After school matching 9 17 356 44 335 2753 
Stage 2 
After student matching 9 17 341 40 221 341 


The pooled LDC elementary school student sample after matching was composed mainly 
of Hispanic and Black students identified as economically disadvantaged (see Table 120). English 
language learners represented one eighth of the sample, while special education students 
represented 15%. Mean performance on the baseline year assessments was about one quarter 
of a standard deviation below the districtwide performance levels in both mathematics and 
ELA. 
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Table 120 


2016-2017 Baseline Characteristics of Treated Students Taught by Cohort 2 Elementary School Teachers 
Participating in LDC in Two Consecutive Years and Comparison Students After Matching 


Treatment group Comparison group 

Student characteristic (n = 341) (n = 341) 
Race/ethnicity 

Hispanic (%) 59.5 59.5 

Black (%) 35.8 35.8 

Asian (%) 3.8 3.8 

White (%) 0.3 0.6 

Other (%) 0.6 0.3 
Female (%) 48.1 48.1 
Special programs status 

Poverty (%) 89.7 89.7 

English language learner (%) 12.9 15.0 

Special education (%) 15.5 15.5 
Student baseline achievement 

Mean baseline year mathematics Z score -0.288 -0.310 

Mean baseline year ELA Z score -0.249 -0.268 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.284 -0.291 

Teacher years of experience 13.8 13.6 
Grade level at baseline year 

Grade 3 at baseline (%) 100 100 


In Table 121, we present results of both the dosage dependent and dosage independent 
models on Cohort 2 elementary school students’ ELA performance in 2018-2019 after being 
taught by teachers with 2 years of LDC experience. Model results for the LDC effect on 
elementary student outcomes are in the positive direction, but are not statistically significant 
for either model. In other words, neither model provided sufficient evidence to conclude that 
elementary students taught by LDC teachers performed better on the ELA test than did their 
matched peers in the comparison group. 
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Table 121 


Effect Estimates of Cohort 2 Elementary School Teachers Participating in LDC in Two Consecutive Years 
on 2018-2019 Smarter Balanced ELA Performance, Dosage-Dependent and Dosage-Independent Models 


Dosage-dependent model Dosage-independent model 


Variables coefficient (SD) coefficient (SD) 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 

Black 

Other 

Poverty 

Female 

English language learner 
Special education 

Teacher Experience 


Baseline peer ELA Z score 


0.094 (0.138) 


-0.038 (0.131) 
-0.165 (0.135) 
0.574 (0.403) 
-0.043 (0.086) 
0.130 (0.054)* 
0.079 (0.082) 
-0.334 (0.079)* 
0.011 (0.007) 
0.121 (0.084) 


0.058 (0.094) 


-0.039 (0.131) 
-0.166 (0.135) 
0.568 (0.403) 
-0.042 (0.085) 
0.131 (0.054)* 
0.079 (0.082) 
-0.335 (0.079)* 
0.012 (0.007) 
0.121 (0.084) 


Baseline year mathematics Z score 0.226 (0.044)* 0.227 (0.043)* 
Baseline year ELA Z score 0.559 (0.048)* 0.559 (0.048)* 


Note. All students were in grade 3 at baseline year. Based on the dosage-dependent model, the average 
treated student received a 0.591 treatment dosage. Because of this, using the dosage-dependent model 
we could estimate an average treatment effect on the treated (ATET)at (0.591 x 0.094) = 0.056. 

*Cl two tailed probability > .95 


The significant effects of the covariates on student performance were similar under the 
two models and were in the expected direction. Baseline ELA performance was the strongest 
predictor and baseline mathematics performance also helped explain the outcome. In addition 
to baseline achievement, two demographic variables helped predict performance: females 
performed at significantly higher levels than males, and special education students performed 
at lower levels than did students not in special education. 


1.8 2017-2018 Cohort 2 Middle School Sample (1 year) 


As shown in Table 122, the resulting Cohort 2 LDC one-year middle school sample included 
10 schools, 31 teachers, and 1,396 students prior to the CEM process. After the CEM student- 
level matching, our final secondary LDC sample was reduced to 1,285 students. Prior to 
matching, the potential comparison sample consisted of 516 schools, 9,867 teachers and 
156,773 students. This comparison sample was substantially reduced during the first stage of 
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matching, which identified up to five schools that closely resembled each of the 10 LDC schools. 
After student-level CEM, a workable analytic comparison sample of 41 schools, 651 teachers, 
and 1,285 students was constructed. 


Table 122 
Before and After Matching Sample Sizes: 2017-2018 Cohort 2 Middle School Analysis 


LDC sample Comparison sample 
Schools Teachers Students Schools Teachers Students 
Stage 1 
Before matching 10 31 1,396 516 9,897 156,773 
After matching 10 31 1,396 50 968 14,651 
Stage 2 
After matching 10 31 1,285 41 651 1,285 


As shown in Table 123 the final Cohort 2 LDC middle school student sample included a 
large combined proportion of Black (51.2%) and Hispanic students (34.0%), and a large majority 
of students who were economically disadvantaged (80.8%). Slightly over one half of the sample 
included students that were in fifth grade (26.4%), and sixth grade (25.1%) in the prior year, 
with the remainder (48.5%) in seventh grade in the prior year. English language learners 
represented about one eighth of this sample, while less than one quarter of the sample 
included special education students (23.8%). In addition, mean performance on the prior year 
assessment for LDC students was substantially lower than district-wide performance levels in 
mathematics and ELA. 
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Table 123 


2016-2017 Baseline Characteristics of Treated Students Taught by Cohort 2 Middle School Teachers in 
2017-2018 and Comparison Students After Matching 


Treatment group Comparison group 
Student characteristic (n = 1,285) (n = 1,285) 

Race/ethnicity 

Hispanic (%) 34.0 34.0 

Black (%) 51.2 51.2 

Asian (%) 11.5 75 

White (%) 2.4 5.1 

Other (%) 0.9 2.2 
Female (%) 48.2 48.2 
Special programs status 

Poverty (%) 80.8 81.6 

English language learner (%) 13.6 11.6 

Special education (%) 23.8 27.8 
Student baseline achievement 

Mean baseline year mathematics Z score -0.540 -0.542 

Mean baseline year ELA Z score -0.409 -0.419 
Class & teacher characteristics 

Mean baseline ELA Z score of current peers -0.434 -0.388 

Teacher years of experience 10.9 10.0 
Grade level at baseline year 

Grade 5 at baseline (%) 26.4 26.4 

Grade 6 at baseline (%) 25.1 25.1 

Grade 7 at baseline (%) 48.5 48.5 


In Table 124 we present results for the students taught by Cohort 2 middle school teachers 
participating in LDC in 2017-2018. Both model results for the LDC effect on student outcomes 
are in the positive direction and did not yield a statistically significant LDC effect. Once again, 
covariates were in the expected direction, with in this case, gender, English Language Learner 
status, and special education status being significant predictors of ELA achievement. 
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Table 124 


Effect Estimates for Cohort 2 LDC Middle School Teachers Participating in LDC for 1 year on 2017-2018 
New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables 


Dosage-dependent model 
coefficient (SD) 


Dosage-independent model 
coefficient (SD) 


Level 2 - LDC teacher treatment 
Level 1 - student characteristics 

Hispanic 

Black 

Asian 

Other 

Poverty 

Female 

English language learner 

Special education 

Grade 6 at baseline 

Grade 7 at baseline 

Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


0.202 (0.108) 


0.107 (0.067) 
0.021 (0.066) 
0.113 (0.074) 
-0.086 (0.116) 
-0.056 (0.032) 
0.130 (0.024)* 
-0.195 (0.044)* 
-0.167 (0.032)* 
0.039 (0.043) 
0.080 (0.039)* 
0.003 (0.004) 
0.092 (0.037)* 
0.206 (0.020)* 
0.653 (0.021)* 


0.078 (0.051) 


0.107 (0.067) 
0.021 (0.066) 
0.112 (0.074) 
-0.088 (0.116) 
-0.055 (0.032) 
0.130 (0.024)* 
-0.196 (0.044)* 
-0.167 (0.032)* 
0.037 (0.043) 
0.081 (0.040)* 
0.003 (0.004) 
0.096 (0.037)* 
0.206 (0.019)* 
0.653 (0.021)* 


Note. For the dosage dependent model, since the average treatment student received a 0.400 
treatment dosage we could estimate an average treatment on the treated (ATET) at 0.40 * 0.202 = 


0.081. 
*Cl two tailed probability > .95 
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1.9 2018-2019 Cohort 2 Middle School Sample (2 years) 


As shown in Table 125, the resulting Cohort 2 LDC middle school sample included nine 
schools, 22 teachers, and 1,033 students prior to the CEM process. After the CEM student-level 
matching, our final secondary LDC sample was reduced to 947 students. Prior to matching, the 
potential comparison sample consisted of 478 schools, 9,919 teachers and 152,128 students. 
This comparison sample was substantially reduced during the first stage of matching, which 
identified up to five schools that closely resembled each of the nine LDC schools. After student- 
level CEM, a workable analytic comparison sample of 37 schools, 547 teachers, and 947 
students was constructed. 


Table 125 
Before and After Matching Sample Sizes: Cohort 2 Middle School (2 years) 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 9 22 1,033 478 9,919 152,128 
After school matching 9 22 1,033 41 1022 12,791 
Stage 2 
After student matching 9 22 947 37 547 947 


As shown in Table 126 the final Cohort 2 LDC middle school student sample included a 
large combined proportion of Black (51.4%) and Hispanic students (35.5%), and a large majority 
of students who were economically disadvantaged (80.6%). English language learners 
represented about one eighth of the sample, and about one quarter of the sample were special 
education students. In addition, mean performance on the prior year assessment for LDC 
students was substantially lower when compared to district-wide performance levels in 
mathematics and ELA. 
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Table 126 


2016-2017 Baseline Characteristics of Treated Students Taught by Cohort 2 Middle School Teachers 
Participating in LDC in Two Consecutive Years and Comparison Students After Matching 


Treatment grou Comparison grou 
Student characteristic te a ” ‘i ae P 

Race/ethnicity 

Hispanic (%) 35.5 35.5 

Black (%) 51.4 51.4 

Asian (%) 9.5 7.6 

White (%) 1.9 43 

Other (%) 1.7 1.2 
Female (%) 45.9 47.2 
Special programs status 

Poverty (%) 80.6 81.9 

English language learner (%) 12.8 13.6 

Special education (%) 24.2 24.2 
Student baseline achievement 

Mean baseline year mathematics Z score -0.526 -0.497 

Mean baseline year ELA Z score -0.423 -0.424 
Class and teacher characteristics 

Mean baseline ELA Z score of current peers -0.449 -0.416 

Teacher years of experience 10.8 10.3 
Grade level at baseline year 

Grade 4 at baseline (%) 21.9 21.9 

Grade 5 at baseline (%) 42.2 42.2 

Grade 6 at baseline (%) 35.9 35.9 


In Table 127 we present analysis results for students taught by Cohort 1 middle school 
teachers who participated in LDC for two consecutive years (2017-2018 and 2018-2019). 
Dosage dependent model results indicate an LDC effect in the negative direction and the 
dosage independent model results was in the positive direction; both effects were very close to 
0 and not statistically significant. Once again, covariates were in the expected direction, with in 
this case, Black, female, and special education status being significant predictors of ELA 
achievement. 
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Table 127 


Effect Estimates of Cohort 2 Middle School Teachers Participating in LDC in Two Consecutive Years on 


2018-2019 New York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables 


Dosage-dependent model 


coefficient (SD) 


coefficient (SD) 


Dosage-independent model 


Level 2 LDC teacher treatment 
Level 1 student characteristics 
Hispanic 

Black 

Other 

Poverty 

Female 

English language learner 
Special education 

Teacher Experience 


Baseline peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 
Grade 5 int baseline 


Grade 6 at baseline 


-0.004 (0.135) 


-0.049 (0.052) 


-0.130 (0.054) * 


0.086 (0.129) 
-0.070 (0.039) 
0.165 (0.030)* 
-0.030 (0.051) 
-0.112 (0.040)* 
0.000 (0.004) 
0.321 (0.039)* 
0.179 (0.024)* 
0.537 (0.026)* 
-0.035 (0.047) 
0.064 (0.048) 


0.061 (0.053) 


-0.051 (0.052) 


-0.131 (0.054) * 


0.083 (0.129) 
-0.070 (0.038) 
0.165 (0.030)* 
-0.030 (0.051) 
-0.112 (0.040)* 
0.000 (0.004) 
0.322 (0.039)* 
0.179 (0.024)* 
0.536 (0.026)* 
-0.034 (0.046) 
0.065 (0.048) 


Note. Asian & White students were combined as reference category to provide adequate sample, and 
total content weeks was not included in this model as it did not reach our tolerance criteria. Based on 
the dosage-dependent model, the average treated student received a 0.336 treatment dosage. Because 


of this, using the dosage-dependent model we could estimate an average treatment effect on the 
treated (ATET)at (0.336 x -0.004) = -0.001. 


*Cl two tailed probability > .95 
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1.10 2018-2019 Cohort 2 Middle School Sample (Newly Joined LDC in 2018- 
2019) 


As shown in Table 128, the resulting Cohort 2 LDC middle school sample included five 
schools, 12 teachers, and 586 students prior to the CEM process. After the CEM student-level 
matching, our final secondary LDC sample was reduced to 547 students. Prior to matching, the 
potential comparison sample consisted of 516 schools, 9,897 teachers and 156,773 students. 
This comparison sample was substantially reduced during the first stage of matching, which 
identified up to five schools that closely resembled each of the five LDC schools. After student- 
level CEM, a workable analytic comparison sample of 22 schools, 298 teachers, and 547 
students was constructed. 


Table 128 
Before and After Matching Sample Sizes: Cohort 2 Middle School Analysis (Newly Joined LDC in 2018— 
2019) 


LDC sample Comparison sample 
Stage Schools Teachers Students Schools Teachers Students 
Stage 1 
Before school matching 5 12 586 516 9,897 156,773 
After school matching 5 12 586 23 451 5,819 
Stage 2 
After student matching 5 12 547 22 298 547 


As shown in Table 129 the final Cohort 2 LDC middle school student was comprised of 
primarily Hispanic and Black students from low socio-economic backgrounds. English language 
learners represented about one eighth of the sample, and about one quarter of the sample 
were special education students. In addition, mean performance on the prior year assessment 
for LDC students was over a half of a standard deviation lower when compared to district-wide 
performance levels in mathematics and ELA. 
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Table 129 


2016-2017 Baseline Characteristics of Students Taught by Cohort 2 Middle School Teachers Who Newly 
Joined LDC in 2018-2019 and Comparison Students After Matching 


Treatment group Comparison group 
Student characteristic (n = 547) (n= 547) 

Race/ethnicity 

Hispanic (%) 29.6 29.6 

Black (%) 62.3 62.3 

Asian (%) 5.3 3.7 

White (%) 1.8 2.9 

Other (%) 0.9 1.5 
Female (%) 50.2 46.1 
Special programs status 

Poverty (%) 86.7 81.4 

English language learner (%) 12.6 12.6 

Special education (%) 26.5 26.5 
Student baseline achievement 

Mean baseline year mathematics Z score -0.598 -0.574 

Mean baseline year ELA Z score -0.529 -0.541 
Class & teacher characteristics 

Mean baseline ELA Z score of current peers -0.553 -0.556 

Teacher years of experience 9.1 10.2 
Grade level at baseline year 

Grade 4 at baseline (%) 24.5 24.5 

Grade 5 at baseline (%) 46.4 46.4 

Grade 6 at baseline (%) 29.1 29.1 


In Table 130 we present results of models that are both dosage dependent and dosage 
independent for the Cohort 2 middle school teachers who newly joined and participated in LDC 
in 2018-2019. Dosage dependent model results indicate an LDC effect in a positive direction 
that approached but did not obtain statistical significance on the student outcome. The dosage 
independent model was in the negative direction and did not yield a statistically significant LDC 
effect. Once again, covariates were in the expected direction, with in this case, Black, female, 
and special education status being significant predictors of ELA achievement. 
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Table 130 


Effect Estimates for Cohort 2 LDC Middle School Teachers Joining LDC in 2018-2019 on 2018-2019 New 
York State ELA Assessment, Dosage-Dependent and Dosage-Independent Models 


Variables Dosage-dependent model Dosage-independent model 
coefficient (SD) coefficient (SD) 


Level 2 - LDC teacher treatment 
Level 1 - student characteristics 
Hispanic 
Black 
Other 
Poverty 
Female 
English language learner 
Special education 
Grade 5 at baseline 
Grade 6 at baseline 
Teacher Experience 


Baseline Peer ELA Z score 


Baseline year mathematics Z score 


Baseline year ELA Z score 


0.029 (0.173) 


-0.151 (0.087) 
-0.224 (0.084)* 
-0.283 (0.192) 
-0.088 (0.053) 
0.186 (0.040)* 
0.048 (0.068) 
-0.125 (0.052)* 
0.155 (0.058)* 
0.189 (0.065)* 
0.013 (0.006)* 
0.233 (0.053)* 
0.183 (0.032)* 
0.556 (0.035)* 


-0.023 (0.067) 


-0.151 (0.087) 
-0.224 (0.084)* 
-0.284 (0.192) 
-0.087 (0.053) 
0.186 (0.040)* 
0.050 (0.068) 
-0.126 (0.052)* 
0.153 (0.058)* 
0.187 (0.065)* 
0.013 (0.006)* 
0.232 (0.053)* 
0.182 (0.032)* 
0.557 (0.035)* 


Note. For the dosage dependent model, since the average treatment student received a 0.323 
treatment dosage we could estimate an average treatment on the treated (ATET) at 0323 * 0.029 = 
0.009. 

*Cl two tailed probability > .95 


1.11 Student Exposure Analysis 


In Table 131, we present the full results of the analysis exploring whether students with 
sustained exposure to LDC over time performed at relatively higher levels than students 
exposed to LDC for the first time in the primary outcome year. The treatment group was split 
into mutually exclusive group, including 1,489 students taught by at least one LDC teacher in 
the prior year, and 1,229 students only exposed to LDC in the outcome year. Separate control 
groups were selected for each mutually exclusive treatment group, and the control group for 
the students with only current exposure served as the reference group for the model. 
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Table 131 


Effect Estimates of Cohort 1 and 2 Elementary and Middle School Teachers Participating in LDC in Two 
Consecutive Years on New York State ELA Assessment, Dosage-Independent Prior Year Subgroups Model 


Variables 


Dosage-Independent Subgroup model 


coefficient (SD) 


Prior and current treatment (n = 1,489) 


Control for prior and current (n = 1,489) 


Current only treatment (n = 1,229) 


Control for current only (n = 1,229) 


Level 1 student characteristics 
Hispanic 
Black 
Asian 
Other 
Poverty 
Female 
English language learner 
Special education 
Teacher Experience 
Baseline peer ELA Z score 
Baseline year mathematics Z score 
Baseline year ELA Z score 
Grade 4 at baseline 
Grade 5 at baseline 
Grade 6 at baseline 


In Cohort 2 schools 


0.092 (0.045)* 
0.046 (0.034) 
-0.007 (0.043) 


Reference Group in Model 


-0.046 (0.042) 
-0.142 (0.044)* 
0.087 (0.053) 
0.162 (0.090) 
-0.091 (0.025)* 
0.173 (0.018)* 
-0.017 (0.029) 
-0.129 (0.024) 
0.004 (0.003) 
0.274 (0.025)* 
0.181 (0.014)* 
0.563 (0.015)* 
-0.037 (0.049) 
-0.038 (0.045) 
0.004 (0.036) 
0.038 (0.033) 


Our primary substantive interest, however, is not in the coefficients as displayed in 
Table 131, but rather in two post-hoc tests of the statistical difference between coefficients, 
displayed in Table 132. Using the MLwiN software5, we obtained the Chi Square value (1 degree 
of freedom) for a test of the difference between two coefficients. The first test examines the 
difference between the coefficient for the treatment group of students exposed to LDC in two 


5 See http://www. bristol.ac.uk/cmm/software/mlwin/ 
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consecutive years, and the coefficient for the control group selected specifically for these 
treatment students. Although the difference (0.046) is in the expected direction, with the 
treatment group coefficient higher than the control group coefficient, the difference (with a p 
value of 0.274) is not significant. The second test examines the difference between the two 
treatment groups: those students with 2 years of LDC exposure and those students with only 
exposure in the outcome year. The difference (0.102) is in the expected direction and 
statistically significant (p = 0.008). Students with 2 years of LDC experience scored significantly 
higher than the students with only 1 year of LDC experience. 


Table 132 
Subgroup results examining exposure of students to LDC over time 


Difference test Difference xX? P value 


Difference between coefficient for students exposed to LDC for one raan vee5 


two consecutive year and coefficient for matched control student 
group 


Difference between coefficient for students exposed to LDC for o0a7 oer coe 


two consecutive year and coefficient for students with exposure 
to LDC only in the outcome year 


In Table 133, we present the full results of the analysis exploring whether students with 
higher LDC dosage in the outcome year performed at relatively higher levels than students 
exposed to lower LDC dosage outcome year. The treatment group was split into mutually 
exclusive group, including 459 students exposed to LDC in half or more of their core content 
area class time, and 2,259 students exposed to LDC in less than half of their core content area 
class time. Separate control groups were selected for each mutually exclusive treatment group, 
and the control group for the student with only current exposure served as the reference group 
for the model. 
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Table 133 

Effect Estimates of Cohort 1 and 2 Elementary and Middle School Teachers Participating in LDC in Two 
Consecutive Years on New York State ELA Assessment, Dosage-Independent Current Year Dosage 
Subgroups Model 


Dosage-Independent Subgroup model 


Variables 


coefficient (SD) 


LDC teacher treatment high dosage (n=459) 


Control for high dosage (n=459) 


LDC teacher treatment low dosage (n=2,259) 


Control for low dosage (n=2,259) 


Level 1 student characteristics 
Hispanic 
Black 

Asian 

Other 
Poverty 
Female 
English language learner 
Special education 

Teacher Experience 
Baseline peer ELA Z score 
Baseline year mathematics Z score 
Baseline year ELA Z score 
Grade 4 at baseline 
Grade 5 at baseline 

Grade 6 at baseline 


In Cohort 2 schools 


-0.010 (0.058) 
0.026 (0.043) 
0.034 (0.038) 


Reference Group in Model 


-0.046 (0.042) 
-0.145 (0.044)* 
0.081 (0.053) 
0.160 (0.090) 
-0.091 (0.025)* 
0.173 (0.018)* 
-0.015 (0.029) 
-0.128 (0.024) 
0.004 (0.003) 
0.179 (0.037)* 
0.276 (0.025)* 
0.563 (0.015)* 
-0.068 (0.049) 
-0.020 (0.046) 
0.027 (0.047) 
0.029 (0.034) 
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Difference tests of interest are displayed in Table 134 below. As can be seen, the 


coefficient for the high dosage treatment group is smaller than the coefficients for both the 
high dosage matched control group and the low dosage treatment group, and these differences 


(-0.036 and -0.044 respectively) are not statistically significant. 


Table 134 
Subgroup results examining level of exposure of students to returning LDC teachers 


Difference test Difference ) P value 
Difference between coefficient for students exposed to returning eae ae poet 
LDC teachers for >=1/2 their total exposure and coefficient for 
matched control student group 
-0.044 0.634 0.426 


Difference between coefficient for students exposed to returning 
LDC teachers for >=1/2 their total exposure and coefficient for 
students with <1/2 their total exposure 


255 


UCLA 


CRESS) 


NATIONAL CENTER FOR RESEARCH ON EVALUATION, 
STANDARDS, AND STUDENT TESTING 


National Center for Research on Evaluation, 
Standards, and Student Testing (CRESST) 


Graduate School of Education & Information Studies 
University of California, Los Angeles 
300 Charles E. Young Drive North 
GSE&IS Bldg., Box 951522 
Los Angeles, CA 90095-1522 


(310) 206-1532 
www.cresst.org 


